Jul 27 17:28:36.307131 ip-10-0-131-74 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 17:28:36.599457 ip-10-0-131-74 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:28:36.599457 ip-10-0-131-74 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 17:28:36.599457 ip-10-0-131-74 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:28:36.599457 ip-10-0-131-74 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 17:28:36.599457 ip-10-0-131-74 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:28:36.600851 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.600727    2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 17:28:36.605160 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605145    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605161    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605166    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605169    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605172    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605176    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605179    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605181    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605184    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605187    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605189    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605192    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605195    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605197    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605200    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605204    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:28:36.605201 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605208    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605216    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605219    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605221    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605224    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605227    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605230    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605240    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605243    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605245    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605248    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605251    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605254    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605256    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605259    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605261    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605264    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605267    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605269    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605272    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:28:36.605578 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605274    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605277    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605280    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605282    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605284    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605287    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605289    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605292    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605294    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605299    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605302    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605305    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605308    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605310    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605314    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605317    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605320    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605323    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605326    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605328    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:28:36.606069 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605331    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605333    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605336    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605338    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605341    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605343    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605346    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605348    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605351    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605354    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605357    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605359    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605361    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605364    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605367    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605369    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605372    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605376    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605378    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605381    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:28:36.606550 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605383    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605386    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605388    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605391    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605393    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605396    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605399    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605402    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605405    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605407    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605410    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605413    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605415    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605418    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605420    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605422    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605425    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605427    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605430    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605433    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:28:36.607040 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605435    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605438    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.605440    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606125    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606131    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606134    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606137    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606140    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606143    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606146    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606148    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606151    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606153    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606156    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606159    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606162    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606164    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606167    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606170    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606173    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:28:36.607614 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606176    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606178    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606181    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606183    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606186    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606189    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606192    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606194    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606197    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606199    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606202    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606204    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606207    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606209    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606212    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606214    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606217    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606219    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606222    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606228    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:28:36.608122 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606231    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606233    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606244    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606247    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606249    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606252    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606255    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606258    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606262    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606266    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606269    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606272    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606275    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606278    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606281    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606283    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606286    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606288    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606291    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:28:36.608606 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606294    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606299    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606302    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606306    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606308    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606312    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606315    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606317    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606320    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606323    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606325    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606328    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606330    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606334    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606337    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606339    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606342    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606344    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606347    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606351    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:28:36.609086 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606354    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606357    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606359    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606363    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606365    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606368    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606370    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606373    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606376    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606378    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606381    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606383    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606386    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606388    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606391    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606393    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606396    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606398    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606401    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:28:36.609564 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606404    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606408    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606411    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606414    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606784    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606788    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606792    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606794    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606797    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606800    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606803    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606805    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606808    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606811    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606813    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606816    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606819    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606821    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606824    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606826    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:28:36.610037 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606829    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606832    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606851    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606855    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606858    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606861    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606865    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606868    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606871    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606874    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606877    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606879    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606882    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606885    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606887    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606890    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606893    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606895    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606898    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:28:36.610548 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606901    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606903    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606906    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606909    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606911    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606914    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606917    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606920    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606923    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606926    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606928    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606931    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606933    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606936    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606938    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606941    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606943    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606946    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606948    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606951    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:28:36.611066 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606953    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606956    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606958    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606961    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606964    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606966    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606968    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606971    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606973    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606976    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606980    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606983    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606986    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606990    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606993    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606996    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.606999    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607002    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607005    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607008    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:28:36.611569 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607011    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607013    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607016    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607018    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607021    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607024    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607026    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607029    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607032    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607034    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607037    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607039    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607042    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607045    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607048    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607050    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607053    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607055    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607058    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607060    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:28:36.612095 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607063    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607065    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607068    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607070    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607140    2571 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607147    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607157    2571 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607164    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607168    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607172    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607176    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607180    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607183    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607186    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607190    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607193    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607196    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607199    2571 flags.go:64] FLAG: --cgroup-root=""
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607202    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607205    2571 flags.go:64] FLAG: --client-ca-file=""
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607208    2571 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607211    2571 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607215    2571 flags.go:64] FLAG: --cluster-domain=""
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607218    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 17:28:36.612589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607222    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607225    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607241    2571 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607245    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607249    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607252    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607256    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607259    2571 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607262    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607265    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607268    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607271    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607277    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607280    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607283    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607288    2571 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607291    2571 flags.go:64] FLAG: --enable-server="true"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607294    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607299    2571 flags.go:64] FLAG: --event-burst="100"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607302    2571 flags.go:64] FLAG: --event-qps="50"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607305    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607308    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607311    2571 flags.go:64] FLAG: --eviction-hard=""
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607314    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 17:28:36.613230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607317    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607320    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607323    2571 flags.go:64] FLAG: --eviction-soft=""
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607326    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607329    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607332    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607335    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607338    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607341    2571 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607344    2571 flags.go:64] FLAG: --feature-gates=""
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607347    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607350    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607353    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607356    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607359    2571 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607362    2571 flags.go:64] FLAG: --help="false"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607365    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-131-74.ec2.internal"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607368    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607371    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607374    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607378    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607381    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607384    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 17:28:36.613932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607390    2571 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607393    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607396    2571 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607399    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607402    2571 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607406    2571 flags.go:64] FLAG: --kube-reserved=""
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607409    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607411    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607414    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607417    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607420    2571 flags.go:64] FLAG: --lock-file=""
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607423    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607426    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607429    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607434    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607437    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607440    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607443    2571 flags.go:64] FLAG: --logging-format="text"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607446    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607449    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607452    2571 flags.go:64] FLAG: --manifest-url=""
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607455    2571 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607463    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607466    2571 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607470    2571 flags.go:64] FLAG: --max-pods="110"
Jul 27 17:28:36.614523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607473    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607476    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607479    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607482    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607486    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607489    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607492    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607500    2571 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607504    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607507    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607510    2571 flags.go:64] FLAG: --pod-cidr=""
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607513    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607519    2571 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607522    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607525    2571 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607528    2571 flags.go:64] FLAG: --port="10250"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607531    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607534    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-010d151a9a9433de1"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607537    2571 flags.go:64] FLAG: --qos-reserved=""
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607540    2571 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607543    2571 flags.go:64] FLAG: --register-node="true"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607546    2571 flags.go:64] FLAG: --register-with-taints=""
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607549    2571 flags.go:64] FLAG: --registry-burst="10"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607552    2571 flags.go:64] FLAG: --registry-qps="5"
Jul 27 17:28:36.615145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607555    2571 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607558    2571 flags.go:64] FLAG: --reserved-memory=""
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607562    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607565    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607568    2571 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607570    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607573    2571 flags.go:64] FLAG: --runonce="false"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607576    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607579    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607582    2571 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607585    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607587    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607591    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607594    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607597    2571 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607601    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607605    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607608    2571 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607611    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607614    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607617    2571 flags.go:64] FLAG: --system-cgroups=""
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607621    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607627    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607630    2571 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607633    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 17:28:36.615929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607637    2571 flags.go:64] FLAG: --tls-min-version=""
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607639    2571 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607642    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607645    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607648    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607651    2571 flags.go:64] FLAG: --v="2"
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607655    2571 flags.go:64] FLAG: --version="false"
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607659    2571 flags.go:64] FLAG: --vmodule=""
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607663    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.607666    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607776    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607779    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607782    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607785    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607789    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607792    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607795    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607798    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607801    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607805    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607808    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607810    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:28:36.616754 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607813    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607817    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607820    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607822    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607825    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607828    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607831    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607846    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607849    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607851    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607854    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607857    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607860    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607862    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607865    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607867    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607870    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607873    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607876    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607878    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:28:36.617347 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607881    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607884    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607886    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607889    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607891    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607894    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607897    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607900    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607902    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607906    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607910    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607914    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607917    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607921    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607924    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607927    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607930    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607933    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607936    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:28:36.617831 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607939    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607942    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607944    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607947    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607949    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607952    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607954    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607957    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607960    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607962    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607965    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607968    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607971    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607973    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607976    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607979    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607982    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607984    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607987    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607990    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:28:36.618325 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607992    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607995    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.607999    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608003    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608006    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608009    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608013    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608016    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608019    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608022    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608025    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608028    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608031    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608033    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608036    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608038    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608041    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608043    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608046    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608049    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:28:36.618817 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608052    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608055    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608059    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608062    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608065    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608067    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608070    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.608072    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.608078    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.614585    2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.614599    2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.615181    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.615192    2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.615348    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:28:36.619324 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.615356    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.615359    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.615362    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.615366    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.615369    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.615380    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616458    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616473    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616479    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616484    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616490    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616496    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616501    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616506    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616510    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616514    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616519    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616524    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616534    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616539    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:28:36.619677 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616543    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616548    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616563    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616569    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616572    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616574    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616577    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616580    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616583    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616610    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616617    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616620    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616623    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616626    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616629    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616631    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616634    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616649    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616654    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:28:36.620162 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616677    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616681    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616784    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616788    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616791    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616794    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616797    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616799    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616802    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616805    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616808    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616810    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616813    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616816    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616820    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616824    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616827    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616830    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616848    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:28:36.620615 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616854    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616856    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616859    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616862    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616865    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616869    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616871    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616874    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616877    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616882    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616884    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616887    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616890    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616893    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616897    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616900    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616903    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616906    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616909    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616912    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:28:36.621190 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616915    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616917    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616920    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616923    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616925    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616928    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616931    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616933    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616936    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616938    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616941    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616944    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616946    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616949    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616952    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616954    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616957    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616959    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616962    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:28:36.621660 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.616964    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.616971    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617093    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617098    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617101    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617104    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617107    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617110    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617114    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617117    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617119    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617122    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617124    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617127    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617130    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:28:36.622131 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617133    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617137    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617141    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617144    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617147    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617150    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617153    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617156    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617158    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617161    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617165    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617167    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617170    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617173    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617175    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617178    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617181    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617184    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617186    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:28:36.622506 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617189    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617191    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617194    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617196    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617199    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617202    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617206    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617209    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617212    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617215    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617217    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617220    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617223    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617225    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617228    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617230    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617233    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617236    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617239    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617241    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:28:36.623013 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617244    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617247    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617249    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617252    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617254    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617257    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617259    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617262    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617264    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617267    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617269    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617272    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617274    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617277    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617279    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617282    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617285    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617288    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617290    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617293    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:28:36.623491 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617295    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617298    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617300    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617303    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617307    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617310    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617313    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617315    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617318    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617321    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617324    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617326    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617329    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617331    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617334    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617336    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617339    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617342    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617344    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:28:36.623980 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617346    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:28:36.624431 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617349    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:28:36.624431 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617352    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:28:36.624431 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617355    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:28:36.624431 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617357    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:28:36.624431 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617360    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:28:36.624431 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617363    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:28:36.624431 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:36.617366    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:28:36.624431 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.617371    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:28:36.624431 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.617481    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 17:28:36.624431 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.619412    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 17:28:36.624431 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.620472    2571 server.go:1019] "Starting client certificate rotation"
Jul 27 17:28:36.624431 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.620573    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 17:28:36.624431 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.620605    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 17:28:36.624767 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.622909    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 17:28:36.624767 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.623995    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 17:28:36.628816 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.628802    2571 log.go:25] "Validated CRI v1 runtime API"
Jul 27 17:28:36.630608 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.630595    2571 log.go:25] "Validated CRI v1 image API"
Jul 27 17:28:36.632955 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.632941    2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 17:28:36.635421 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.635398    2571 fs.go:135] Filesystem UUIDs: map[6b22bc3a-cf6c-4613-ac82-ac3160914969:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b6d94eb8-52e3-49d9-bdd0-b142dfaf1508:/dev/nvme0n1p3]
Jul 27 17:28:36.635478 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.635421    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 17:28:36.640938 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.640816    2571 manager.go:216] Machine: {Timestamp:2026-07-27 17:28:36.640343357 +0000 UTC m=+0.242740477 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097696 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28f4e43e2c822a470e30cb6e031539 SystemUUID:ec28f4e4-3e2c-822a-470e-30cb6e031539 BootID:933904f4-da8f-4952-aa4e-32fec152f71b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:55:f4:04:cf:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:55:f4:04:cf:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:34:b2:23:13:af Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 17:28:36.640938 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.640933    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 17:28:36.641046 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.641032    2571 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 17:28:36.643296 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.643277    2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 17:28:36.643444 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.643297    2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-74.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 17:28:36.643882 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.643872    2571 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 17:28:36.643913 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.643884    2571 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 17:28:36.643913 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.643891    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 17:28:36.644456 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.644446    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 17:28:36.645239 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.645228    2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 17:28:36.647830 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.647820    2571 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 17:28:36.648059 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.648048    2571 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 17:28:36.648657 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.648646    2571 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 17:28:36.648692 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.648669    2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 17:28:36.648692 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.648683    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 17:28:36.648692 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.648692    2571 kubelet.go:398] "Adding apiserver pod source"
Jul 27 17:28:36.648802 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.648709    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 17:28:36.649278 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.649261    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 17:28:36.649321 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.649297    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 17:28:36.652064 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.651525    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 17:28:36.652064 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.651700    2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 17:28:36.653543 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.653528    2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 17:28:36.653603 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.653558    2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 17:28:36.656142 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656128    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 17:28:36.656142 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656146    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 17:28:36.656252 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656153    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 17:28:36.656252 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656159    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 17:28:36.656252 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656168    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 17:28:36.656252 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656173    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 17:28:36.656252 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656182    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 17:28:36.656252 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656187    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 17:28:36.656252 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656194    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 17:28:36.656252 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656200    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 17:28:36.656252 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656208    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 17:28:36.656252 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656213    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 17:28:36.656788 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656773    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 17:28:36.656788 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.656783    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 17:28:36.660160 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.660147    2571 server.go:1268] "Started kubelet"
Jul 27 17:28:36.660257 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.660236    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 17:28:36.660665 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.660615    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 17:28:36.660748 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.660687    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 17:28:36.661017 ip-10-0-131-74 systemd[1]: Started Kubernetes Kubelet.
Jul 27 17:28:36.662518 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.662502    2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 17:28:36.662615 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.662601    2571 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 17:28:36.663192 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.663166    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 17:28:36.663286 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.663205    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-74.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 17:28:36.663373 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.663347    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 17:28:36.665986 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.665950    2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 17:28:36.667262 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.666550    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-74.ec2.internal.18c635ac5a084266  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-74.ec2.internal,UID:ip-10-0-131-74.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-74.ec2.internal,},FirstTimestamp:2026-07-27 17:28:36.660118118 +0000 UTC m=+0.262515239,LastTimestamp:2026-07-27 17:28:36.660118118 +0000 UTC m=+0.262515239,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-74.ec2.internal,}"
Jul 27 17:28:36.668927 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.668911    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 17:28:36.668989 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.668916    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 17:28:36.669047 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.668995    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-74.ec2.internal\" not found"
Jul 27 17:28:36.669620 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.669598    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 17:28:36.669849 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.669819    2571 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 17:28:36.669918 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.669853    2571 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 17:28:36.669918 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.669874    2571 factory.go:55] Registering systemd factory
Jul 27 17:28:36.669918 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.669912    2571 factory.go:223] Registration of the systemd container factory successfully
Jul 27 17:28:36.670041 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.669984    2571 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 17:28:36.670041 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.669991    2571 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 17:28:36.671259 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.670478    2571 factory.go:153] Registering CRI-O factory
Jul 27 17:28:36.671259 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.670499    2571 factory.go:223] Registration of the crio container factory successfully
Jul 27 17:28:36.671259 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.670566    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 17:28:36.671259 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.670587    2571 factory.go:103] Registering Raw factory
Jul 27 17:28:36.671259 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.670602    2571 manager.go:1196] Started watching for new ooms in manager
Jul 27 17:28:36.671259 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.671082    2571 manager.go:349] Starting recovery of all containers
Jul 27 17:28:36.672039 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.672016    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mmcf7"
Jul 27 17:28:36.673207 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.673181    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 17:28:36.680319 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.680296    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mmcf7"
Jul 27 17:28:36.682304 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.682287    2571 manager.go:354] Recovery completed
Jul 27 17:28:36.682826 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.682373    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 17:28:36.687679 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.687664    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:36.689741 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.689725    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:36.689810 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.689754    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:36.689810 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.689769    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:36.690225 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.690205    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 17:28:36.690225 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.690213    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:36.690225 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.690219    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 17:28:36.690371 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.690248    2571 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 17:28:36.690965 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.690948    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:36.691032 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.690977    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:36.691032 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.690987    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:36.691910 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.691800    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-74.ec2.internal.18c635ac5bcc3d77  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-74.ec2.internal,UID:ip-10-0-131-74.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-74.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-74.ec2.internal,},FirstTimestamp:2026-07-27 17:28:36.689739127 +0000 UTC m=+0.292136247,LastTimestamp:2026-07-27 17:28:36.689739127 +0000 UTC m=+0.292136247,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-74.ec2.internal,}"
Jul 27 17:28:36.693533 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.693521    2571 policy_none.go:49] "None policy: Start"
Jul 27 17:28:36.693609 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.693538    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 17:28:36.693609 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.693551    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 17:28:36.695560 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.695548    2571 policy_none.go:47] "Start"
Jul 27 17:28:36.731495 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.731379    2571 manager.go:337] "Starting Device Plugin manager"
Jul 27 17:28:36.731581 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.731540    2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 17:28:36.731581 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.731553    2571 server.go:87] "Starting device plugin registration server"
Jul 27 17:28:36.731767 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.731754    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 17:28:36.731806 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.731769    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 17:28:36.731903 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.731886    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 17:28:36.731986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.731976    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 17:28:36.731986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.731982    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 17:28:36.732441 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.732421    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 17:28:36.732505 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.732463    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-74.ec2.internal\" not found"
Jul 27 17:28:36.831923 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.831902    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:36.834140 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.832985    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:36.834140 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.833025    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:36.834140 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.833040    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:36.834140 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.833073    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-74.ec2.internal"
Jul 27 17:28:36.834140 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.833723    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 17:28:36.834965 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.834951    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 17:28:36.835013 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.834992    2571 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 17:28:36.835058 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.835018    2571 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 17:28:36.835058 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.835053    2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 17:28:36.836663 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.836647    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:28:36.840287 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.840272    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-74.ec2.internal"
Jul 27 17:28:36.840368 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.840292    2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-74.ec2.internal\": node \"ip-10-0-131-74.ec2.internal\" not found"
Jul 27 17:28:36.881520 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.881472    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-74.ec2.internal\" not found"
Jul 27 17:28:36.936106 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.936085    2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal"]
Jul 27 17:28:36.936165 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.936155    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:36.938306 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.938292    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:36.938358 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.938318    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:36.938358 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.938330    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:36.940653 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.940641    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:36.940771 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.940759    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:36.940808 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.940786    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:36.943039 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.943020    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:36.943129 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.943048    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:36.943129 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.943062    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:36.943129 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.943047    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:36.943235 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.943141    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:36.943235 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.943158    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:36.945300 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.945286    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:36.945342 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.945309    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:36.945947 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.945933    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:36.946011 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.945960    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:36.946011 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.945975    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:36.969427 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.969410    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-74.ec2.internal\" not found" node="ip-10-0-131-74.ec2.internal"
Jul 27 17:28:36.971435 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.971400    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f6587cb3b426ccd7ab481b25ab3a2091-config\") pod \"kube-apiserver-proxy-ip-10-0-131-74.ec2.internal\" (UID: \"f6587cb3b426ccd7ab481b25ab3a2091\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:36.971529 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.971434    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea5219289a853fa387fd5432a83f50ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal\" (UID: \"ea5219289a853fa387fd5432a83f50ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:36.971529 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:36.971460    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea5219289a853fa387fd5432a83f50ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal\" (UID: \"ea5219289a853fa387fd5432a83f50ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:36.973690 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.973676    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-74.ec2.internal\" not found" node="ip-10-0-131-74.ec2.internal"
Jul 27 17:28:36.982411 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:36.982392    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-74.ec2.internal\" not found"
Jul 27 17:28:37.072081 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:37.072060    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea5219289a853fa387fd5432a83f50ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal\" (UID: \"ea5219289a853fa387fd5432a83f50ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:37.072169 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:37.072086    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea5219289a853fa387fd5432a83f50ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal\" (UID: \"ea5219289a853fa387fd5432a83f50ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:37.072169 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:37.072104    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f6587cb3b426ccd7ab481b25ab3a2091-config\") pod \"kube-apiserver-proxy-ip-10-0-131-74.ec2.internal\" (UID: \"f6587cb3b426ccd7ab481b25ab3a2091\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:37.072169 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:37.072157    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea5219289a853fa387fd5432a83f50ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal\" (UID: \"ea5219289a853fa387fd5432a83f50ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:37.072295 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:37.072167    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea5219289a853fa387fd5432a83f50ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal\" (UID: \"ea5219289a853fa387fd5432a83f50ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:37.072295 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:37.072225    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f6587cb3b426ccd7ab481b25ab3a2091-config\") pod \"kube-apiserver-proxy-ip-10-0-131-74.ec2.internal\" (UID: \"f6587cb3b426ccd7ab481b25ab3a2091\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:37.083044 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:37.083025    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-74.ec2.internal\" not found"
Jul 27 17:28:37.183872 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:37.183790    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-74.ec2.internal\" not found"
Jul 27 17:28:37.272155 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:37.272126    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:37.277445 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:37.276865    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:37.284897 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:37.284880    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-74.ec2.internal\" not found"
Jul 27 17:28:37.385355 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:37.385328    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-74.ec2.internal\" not found"
Jul 27 17:28:39.043610 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:39.043538    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-74.ec2.internal\" not found"
Jul 27 17:28:39.144023 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:39.143995    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-74.ec2.internal\" not found"
Jul 27 17:28:39.178170 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:39.178145    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 17:28:39.178666 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:39.178282    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 17:28:39.232912 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:39.232887    2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:28:39.245131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:39.245107    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 17:23:38 +0000 UTC" deadline="2028-04-10 00:07:06.85086225 +0000 UTC"
Jul 27 17:28:39.245180 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:39.245131    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14934h38m27.605732994s"
Jul 27 17:28:39.327229 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:39.327150    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:39.338934 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:39.338915    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 17:28:39.340217 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:39.340204    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal"
Jul 27 17:28:39.348859 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:39.348842    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 17:28:39.462655 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:39.462629    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:28:39.596521 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:39.596438    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:28:39.659605 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:39.659568    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5219289a853fa387fd5432a83f50ac.slice/crio-d7c24e2a565feab08e2f99ca4176cf9e3fdaf1f6a9370d67294ae3483963d615 WatchSource:0}: Error finding container d7c24e2a565feab08e2f99ca4176cf9e3fdaf1f6a9370d67294ae3483963d615: Status 404 returned error can't find the container with id d7c24e2a565feab08e2f99ca4176cf9e3fdaf1f6a9370d67294ae3483963d615
Jul 27 17:28:39.663276 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:39.663258    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:28:40.207576 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.207546    2571 apiserver.go:52] "Watching apiserver"
Jul 27 17:28:40.216150 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.216129    2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 17:28:40.216526 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.216479    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-8lrn8","openshift-network-diagnostics/network-check-target-7qfvm","openshift-network-operator/iptables-alerter-2vfdv","openshift-ovn-kubernetes/ovnkube-node-74rhp","kube-system/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx","openshift-cluster-node-tuning-operator/tuned-k5v8t","openshift-image-registry/node-ca-ckvrn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal","openshift-multus/multus-additional-cni-plugins-xdrd5","openshift-multus/multus-njw42","openshift-multus/network-metrics-daemon-vtptl","kube-system/global-pull-secret-syncer-8kxfj","kube-system/konnectivity-agent-cq6fr"]
Jul 27 17:28:40.220027 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.220006    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.222186 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.222168    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.222625 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.222610    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:28:40.222913 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.222899    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 17:28:40.223081 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.223065    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m94gx\""
Jul 27 17:28:40.224294 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.224271    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2vfdv"
Jul 27 17:28:40.224588 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.224568    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hpmw9\""
Jul 27 17:28:40.224689 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.224657    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 17:28:40.224689 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.224670    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 17:28:40.224804 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.224704    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 17:28:40.224904 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.224890    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 17:28:40.226383 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.226370    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 17:28:40.226666 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.226630    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 17:28:40.226764 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.226725    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.226764 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.226736    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:28:40.226866 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.226723    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c7m4d\""
Jul 27 17:28:40.226999 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.226982    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 17:28:40.228916 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.228896    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cq6fr"
Jul 27 17:28:40.229244 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.229226    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 17:28:40.229330 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.229265    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2g5mv\""
Jul 27 17:28:40.229458 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.229438    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 17:28:40.229568 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.229475    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 17:28:40.230517 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.230485    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 17:28:40.230581 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.230525    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 17:28:40.230581 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.230535    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 17:28:40.231345 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.231333    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.231954 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.231759    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 17:28:40.233622 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.232209    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 17:28:40.233622 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.232247    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fnh6h\""
Jul 27 17:28:40.233762 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.233688    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 17:28:40.233762 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.233727    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tcmmz\""
Jul 27 17:28:40.234570 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.234055    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 17:28:40.234570 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.234061    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 17:28:40.235278 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.235258    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ckvrn"
Jul 27 17:28:40.237329 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237304    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-kubelet\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.237413 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237339    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-cni-netd\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.237413 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237363    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/926eec82-2a89-4e47-995c-0de36ce09d69-ovn-node-metrics-cert\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.237532 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237413    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-sysconfig\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.237532 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237446    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-sys\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.237532 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237468    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-sysctl-conf\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.237532 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237490    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-systemd\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.237532 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237527    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-var-lib-kubelet\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.237750 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237565    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-host\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.237750 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237581    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 17:28:40.237750 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237598    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-tuned\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.237750 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237602    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:40.237750 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237636    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-os-release\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.237750 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.237670    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:28:40.237750 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237677    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-run-netns\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.237750 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237735    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-node-log\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237762    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-systemd-units\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237779    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-run-netns\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237804    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237838    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/926eec82-2a89-4e47-995c-0de36ce09d69-env-overrides\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237861    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-cni-binary-copy\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.238049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237879    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-run-openvswitch\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237895    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-log-socket\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237910    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-run-ovn-kubernetes\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237937    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-var-lib-kubelet\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.238049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237958    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae0eab36-cd38-40c9-b640-92692454514f-host-slash\") pod \"iptables-alerter-2vfdv\" (UID: \"ae0eab36-cd38-40c9-b640-92692454514f\") " pod="openshift-network-operator/iptables-alerter-2vfdv"
Jul 27 17:28:40.238049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.237989    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-slash\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238018    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk4dq\" (UniqueName: \"kubernetes.io/projected/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-kube-api-access-wk4dq\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.238049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238042    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-multus-conf-dir\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238071    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/926eec82-2a89-4e47-995c-0de36ce09d69-ovnkube-config\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238096    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-var-lib-openvswitch\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238118    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-modprobe-d\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238141    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-multus-cni-dir\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238163    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-cnibin\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238186    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h6qz\" (UniqueName: \"kubernetes.io/projected/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-kube-api-access-9h6qz\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238207    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-run-systemd\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-kubernetes\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238278    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-run\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238304    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-var-lib-cni-bin\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238316    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238327    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-run-ovn\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238348    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-25q8b\""
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238349    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-hostroot\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238366    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238400    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-cni-bin\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238451    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa-konnectivity-ca\") pod \"konnectivity-agent-cq6fr\" (UID: \"7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa\") " pod="kube-system/konnectivity-agent-cq6fr"
Jul 27 17:28:40.238586 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238477    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-sysctl-d\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238517    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-lib-modules\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238537    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-run-k8s-cni-cncf-io\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238551    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-etc-kubernetes\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238565    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae0eab36-cd38-40c9-b640-92692454514f-iptables-alerter-script\") pod \"iptables-alerter-2vfdv\" (UID: \"ae0eab36-cd38-40c9-b640-92692454514f\") " pod="openshift-network-operator/iptables-alerter-2vfdv"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238582    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-etc-openvswitch\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238624    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/926eec82-2a89-4e47-995c-0de36ce09d69-ovnkube-script-lib\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238670    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n97b2\" (UniqueName: \"kubernetes.io/projected/926eec82-2a89-4e47-995c-0de36ce09d69-kube-api-access-n97b2\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238686    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa-agent-certs\") pod \"konnectivity-agent-cq6fr\" (UID: \"7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa\") " pod="kube-system/konnectivity-agent-cq6fr"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238701    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-multus-socket-dir-parent\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238730    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-multus-daemon-config\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238759    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92gks\" (UniqueName: \"kubernetes.io/projected/ae0eab36-cd38-40c9-b640-92692454514f-kube-api-access-92gks\") pod \"iptables-alerter-2vfdv\" (UID: \"ae0eab36-cd38-40c9-b640-92692454514f\") " pod="openshift-network-operator/iptables-alerter-2vfdv"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238783    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-tmp\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238808    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-system-cni-dir\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238831    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-var-lib-cni-multus\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.239131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.238850    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-run-multus-certs\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.239677 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.239259    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 17:28:40.239884 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.239869    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:40.240066 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.239975    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:28:40.242195 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.242181    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.244279 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.244259    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 17:28:40.244279 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.244273    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rd5k7\""
Jul 27 17:28:40.244490 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.244476    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 17:28:40.244567 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.244554    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8lrn8"
Jul 27 17:28:40.246713 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.246697    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 17:28:40.246713 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.246710    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fljwx\""
Jul 27 17:28:40.246948 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.246931    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 17:28:40.247134 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.247043    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:40.247134 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.247099    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:28:40.260532 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.260516    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6fl66"
Jul 27 17:28:40.268288 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.268270    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6fl66"
Jul 27 17:28:40.328348 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.328325    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 17:28:40.339125 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339104    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-sysconfig\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.339217 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339128    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-sys\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.339217 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339148    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-dbus\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:40.339217 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339164    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:40.339217 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339179    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-sysctl-conf\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.339217 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339201    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-systemd\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339225    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-var-lib-kubelet\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339231    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-sys\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339235    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-sysconfig\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339268    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-host\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339282    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-systemd\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339327    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-sysctl-conf\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339336    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-tuned\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339344    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-host\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339339    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-var-lib-kubelet\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339361    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-os-release\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339385    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-run-netns\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339397    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-os-release\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339411    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-node-log\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339421    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-run-netns\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339433    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-systemd-units\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.339440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339447    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-node-log\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339454    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-run-netns\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339471    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-systemd-units\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339481    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339509    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-run-netns\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339524    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/926eec82-2a89-4e47-995c-0de36ce09d69-env-overrides\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339546    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339550    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/367e4815-f156-4241-88f1-b8e84258fd04-serviceca\") pod \"node-ca-ckvrn\" (UID: \"367e4815-f156-4241-88f1-b8e84258fd04\") " pod="openshift-image-registry/node-ca-ckvrn"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339575    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2s9\" (UniqueName: \"kubernetes.io/projected/367e4815-f156-4241-88f1-b8e84258fd04-kube-api-access-vx2s9\") pod \"node-ca-ckvrn\" (UID: \"367e4815-f156-4241-88f1-b8e84258fd04\") " pod="openshift-image-registry/node-ca-ckvrn"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339600    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf8670f0-936b-46cb-af2d-f0465743f16d-cnibin\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339630    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339627    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-cni-binary-copy\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339664    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-run-openvswitch\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339683    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-log-socket\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339702    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-run-ovn-kubernetes\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339749    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-log-socket\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339756    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-run-openvswitch\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339785    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-run-ovn-kubernetes\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339795    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339835    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g4sl\" (UniqueName: \"kubernetes.io/projected/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-kube-api-access-9g4sl\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339859    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf8670f0-936b-46cb-af2d-f0465743f16d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339881    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-var-lib-kubelet\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339907    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae0eab36-cd38-40c9-b640-92692454514f-host-slash\") pod \"iptables-alerter-2vfdv\" (UID: \"ae0eab36-cd38-40c9-b640-92692454514f\") " pod="openshift-network-operator/iptables-alerter-2vfdv"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339912    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/926eec82-2a89-4e47-995c-0de36ce09d69-env-overrides\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339922    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-slash\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339948    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcmlz\" (UniqueName: \"kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz\") pod \"network-check-target-7qfvm\" (UID: \"6bb08eb3-0b94-49f8-8c53-650e6ae67145\") " pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339962    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-var-lib-kubelet\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339973    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf8670f0-936b-46cb-af2d-f0465743f16d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339982    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-slash\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339983    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae0eab36-cd38-40c9-b640-92692454514f-host-slash\") pod \"iptables-alerter-2vfdv\" (UID: \"ae0eab36-cd38-40c9-b640-92692454514f\") " pod="openshift-network-operator/iptables-alerter-2vfdv"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.339998    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf8670f0-936b-46cb-af2d-f0465743f16d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340020    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf8670f0-936b-46cb-af2d-f0465743f16d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340037    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk4dq\" (UniqueName: \"kubernetes.io/projected/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-kube-api-access-wk4dq\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.340735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340051    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-multus-conf-dir\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340073    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/926eec82-2a89-4e47-995c-0de36ce09d69-ovnkube-config\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340092    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-multus-conf-dir\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340095    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340123    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-registration-dir\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340132    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-cni-binary-copy\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340150    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-var-lib-openvswitch\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340179    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-795z9\" (UniqueName: \"kubernetes.io/projected/cf8670f0-936b-46cb-af2d-f0465743f16d-kube-api-access-795z9\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340185    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-var-lib-openvswitch\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340201    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-modprobe-d\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340228    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-multus-cni-dir\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340284    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-cnibin\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340306    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h6qz\" (UniqueName: \"kubernetes.io/projected/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-kube-api-access-9h6qz\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340328    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-run-systemd\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340346    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-multus-cni-dir\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340351    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-cnibin\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340356    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-modprobe-d\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.341422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340352    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-device-dir\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340376    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-run-systemd\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340420    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf8670f0-936b-46cb-af2d-f0465743f16d-system-cni-dir\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340447    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f50b4a82-db6d-409a-8c83-eb229b3e2e13-tmp-dir\") pod \"node-resolver-8lrn8\" (UID: \"f50b4a82-db6d-409a-8c83-eb229b3e2e13\") " pod="openshift-dns/node-resolver-8lrn8"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340464    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-kubernetes\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340478    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-run\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340509    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-var-lib-cni-bin\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340531    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/926eec82-2a89-4e47-995c-0de36ce09d69-ovnkube-config\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340541    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-kubernetes\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340551    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-run\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340535    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-run-ovn\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340568    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-var-lib-cni-bin\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340567    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-run-ovn\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340628    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-hostroot\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340647    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-cni-bin\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340661    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa-konnectivity-ca\") pod \"konnectivity-agent-cq6fr\" (UID: \"7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa\") " pod="kube-system/konnectivity-agent-cq6fr"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340677    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/367e4815-f156-4241-88f1-b8e84258fd04-host\") pod \"node-ca-ckvrn\" (UID: \"367e4815-f156-4241-88f1-b8e84258fd04\") " pod="openshift-image-registry/node-ca-ckvrn"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340707    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-sysctl-d\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.342184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340718    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-cni-bin\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340733    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-lib-modules\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340740    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-hostroot\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340757    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-socket-dir\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340781    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5c8l\" (UniqueName: \"kubernetes.io/projected/f50b4a82-db6d-409a-8c83-eb229b3e2e13-kube-api-access-r5c8l\") pod \"node-resolver-8lrn8\" (UID: \"f50b4a82-db6d-409a-8c83-eb229b3e2e13\") " pod="openshift-dns/node-resolver-8lrn8"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340812    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-run-k8s-cni-cncf-io\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340814    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-sysctl-d\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340835    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-etc-kubernetes\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340849    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-lib-modules\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340853    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-run-k8s-cni-cncf-io\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340859    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae0eab36-cd38-40c9-b640-92692454514f-iptables-alerter-script\") pod \"iptables-alerter-2vfdv\" (UID: \"ae0eab36-cd38-40c9-b640-92692454514f\") " pod="openshift-network-operator/iptables-alerter-2vfdv"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340882    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-etc-openvswitch\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340889    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-etc-kubernetes\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340899    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/926eec82-2a89-4e47-995c-0de36ce09d69-ovnkube-script-lib\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340917    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n97b2\" (UniqueName: \"kubernetes.io/projected/926eec82-2a89-4e47-995c-0de36ce09d69-kube-api-access-n97b2\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340940    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340951    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-etc-openvswitch\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.342986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340963    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf8670f0-936b-46cb-af2d-f0465743f16d-os-release\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.340987    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa-agent-certs\") pod \"konnectivity-agent-cq6fr\" (UID: \"7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa\") " pod="kube-system/konnectivity-agent-cq6fr"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341010    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f50b4a82-db6d-409a-8c83-eb229b3e2e13-hosts-file\") pod \"node-resolver-8lrn8\" (UID: \"f50b4a82-db6d-409a-8c83-eb229b3e2e13\") " pod="openshift-dns/node-resolver-8lrn8"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341032    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-multus-socket-dir-parent\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341056    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-multus-daemon-config\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341080    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92gks\" (UniqueName: \"kubernetes.io/projected/ae0eab36-cd38-40c9-b640-92692454514f-kube-api-access-92gks\") pod \"iptables-alerter-2vfdv\" (UID: \"ae0eab36-cd38-40c9-b640-92692454514f\") " pod="openshift-network-operator/iptables-alerter-2vfdv"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341105    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-sys-fs\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341128    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-tmp\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341155    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-system-cni-dir\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341178    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-var-lib-cni-multus\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341222    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-run-multus-certs\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341238    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa-konnectivity-ca\") pod \"konnectivity-agent-cq6fr\" (UID: \"7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa\") " pod="kube-system/konnectivity-agent-cq6fr"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341248    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-kubelet\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341271    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-cni-netd\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341290    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl5x8\" (UniqueName: \"kubernetes.io/projected/29c8b524-a2af-4426-a4c7-dc102e6154ac-kube-api-access-pl5x8\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341306    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-system-cni-dir\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341318    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-kubelet-config\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:40.343706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341327    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-run-multus-certs\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.344165 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341349    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/926eec82-2a89-4e47-995c-0de36ce09d69-ovn-node-metrics-cert\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.344165 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341371    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-cni-netd\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.344165 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341389    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-host-var-lib-cni-multus\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.344165 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341464    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ae0eab36-cd38-40c9-b640-92692454514f-iptables-alerter-script\") pod \"iptables-alerter-2vfdv\" (UID: \"ae0eab36-cd38-40c9-b640-92692454514f\") " pod="openshift-network-operator/iptables-alerter-2vfdv"
Jul 27 17:28:40.344165 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341536    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/926eec82-2a89-4e47-995c-0de36ce09d69-host-kubelet\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.344165 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341607    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-multus-socket-dir-parent\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.344165 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341668    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/926eec82-2a89-4e47-995c-0de36ce09d69-ovnkube-script-lib\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.344165 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.341989    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-multus-daemon-config\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.344165 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.343304    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-tmp\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.344165 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.343365    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-etc-tuned\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.344165 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.343517    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa-agent-certs\") pod \"konnectivity-agent-cq6fr\" (UID: \"7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa\") " pod="kube-system/konnectivity-agent-cq6fr"
Jul 27 17:28:40.344165 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.343559    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/926eec82-2a89-4e47-995c-0de36ce09d69-ovn-node-metrics-cert\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.350252 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.350227    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h6qz\" (UniqueName: \"kubernetes.io/projected/7aa19a15-4b11-4bb9-a47a-be6eb5d581db-kube-api-access-9h6qz\") pod \"multus-njw42\" (UID: \"7aa19a15-4b11-4bb9-a47a-be6eb5d581db\") " pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.350352 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.350299    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92gks\" (UniqueName: \"kubernetes.io/projected/ae0eab36-cd38-40c9-b640-92692454514f-kube-api-access-92gks\") pod \"iptables-alerter-2vfdv\" (UID: \"ae0eab36-cd38-40c9-b640-92692454514f\") " pod="openshift-network-operator/iptables-alerter-2vfdv"
Jul 27 17:28:40.350479 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.350461    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk4dq\" (UniqueName: \"kubernetes.io/projected/55bb8c6b-57ef-45fa-bfd9-8c01892ca798-kube-api-access-wk4dq\") pod \"tuned-k5v8t\" (UID: \"55bb8c6b-57ef-45fa-bfd9-8c01892ca798\") " pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.350674 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.350658    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n97b2\" (UniqueName: \"kubernetes.io/projected/926eec82-2a89-4e47-995c-0de36ce09d69-kube-api-access-n97b2\") pod \"ovnkube-node-74rhp\" (UID: \"926eec82-2a89-4e47-995c-0de36ce09d69\") " pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.373921 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.373902    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:28:40.396758 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.396725    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal" event={"ID":"ea5219289a853fa387fd5432a83f50ac","Type":"ContainerStarted","Data":"d7c24e2a565feab08e2f99ca4176cf9e3fdaf1f6a9370d67294ae3483963d615"}
Jul 27 17:28:40.441571 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441547    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl5x8\" (UniqueName: \"kubernetes.io/projected/29c8b524-a2af-4426-a4c7-dc102e6154ac-kube-api-access-pl5x8\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.441668 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441576    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-kubelet-config\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:40.441668 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-dbus\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:40.441668 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441629    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:40.441815 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441684    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-kubelet-config\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:40.441815 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441690    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/367e4815-f156-4241-88f1-b8e84258fd04-serviceca\") pod \"node-ca-ckvrn\" (UID: \"367e4815-f156-4241-88f1-b8e84258fd04\") " pod="openshift-image-registry/node-ca-ckvrn"
Jul 27 17:28:40.441815 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.441722    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:40.441815 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441741    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2s9\" (UniqueName: \"kubernetes.io/projected/367e4815-f156-4241-88f1-b8e84258fd04-kube-api-access-vx2s9\") pod \"node-ca-ckvrn\" (UID: \"367e4815-f156-4241-88f1-b8e84258fd04\") " pod="openshift-image-registry/node-ca-ckvrn"
Jul 27 17:28:40.441815 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441759    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-dbus\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:40.441815 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf8670f0-936b-46cb-af2d-f0465743f16d-cnibin\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.441815 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.441783    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret podName:cbe546c0-bfc4-4692-a40b-3f0d22c0760d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:40.941764122 +0000 UTC m=+2.986502197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret") pod "global-pull-secret-syncer-8kxfj" (UID: "cbe546c0-bfc4-4692-a40b-3f0d22c0760d") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:40.441815 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441808    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf8670f0-936b-46cb-af2d-f0465743f16d-cnibin\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441848    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441878    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g4sl\" (UniqueName: \"kubernetes.io/projected/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-kube-api-access-9g4sl\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441902    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf8670f0-936b-46cb-af2d-f0465743f16d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441929    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmlz\" (UniqueName: \"kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz\") pod \"network-check-target-7qfvm\" (UID: \"6bb08eb3-0b94-49f8-8c53-650e6ae67145\") " pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.441941    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441955    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf8670f0-936b-46cb-af2d-f0465743f16d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.441979    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf8670f0-936b-46cb-af2d-f0465743f16d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.441997    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs podName:50fb06f3-ab71-43cc-aa9c-58ba1b7651d6 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:40.94197961 +0000 UTC m=+2.986717683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs") pod "network-metrics-daemon-vtptl" (UID: "50fb06f3-ab71-43cc-aa9c-58ba1b7651d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442019    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf8670f0-936b-46cb-af2d-f0465743f16d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442040    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442063    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-registration-dir\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442090    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-795z9\" (UniqueName: \"kubernetes.io/projected/cf8670f0-936b-46cb-af2d-f0465743f16d-kube-api-access-795z9\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442098    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/367e4815-f156-4241-88f1-b8e84258fd04-serviceca\") pod \"node-ca-ckvrn\" (UID: \"367e4815-f156-4241-88f1-b8e84258fd04\") " pod="openshift-image-registry/node-ca-ckvrn"
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442105    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf8670f0-936b-46cb-af2d-f0465743f16d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.442120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442118    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442122    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-device-dir\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442143    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-registration-dir\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442162    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf8670f0-936b-46cb-af2d-f0465743f16d-system-cni-dir\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442187    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f50b4a82-db6d-409a-8c83-eb229b3e2e13-tmp-dir\") pod \"node-resolver-8lrn8\" (UID: \"f50b4a82-db6d-409a-8c83-eb229b3e2e13\") " pod="openshift-dns/node-resolver-8lrn8"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442165    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-device-dir\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442206    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf8670f0-936b-46cb-af2d-f0465743f16d-system-cni-dir\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442217    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/367e4815-f156-4241-88f1-b8e84258fd04-host\") pod \"node-ca-ckvrn\" (UID: \"367e4815-f156-4241-88f1-b8e84258fd04\") " pod="openshift-image-registry/node-ca-ckvrn"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442243    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-socket-dir\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442267    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5c8l\" (UniqueName: \"kubernetes.io/projected/f50b4a82-db6d-409a-8c83-eb229b3e2e13-kube-api-access-r5c8l\") pod \"node-resolver-8lrn8\" (UID: \"f50b4a82-db6d-409a-8c83-eb229b3e2e13\") " pod="openshift-dns/node-resolver-8lrn8"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442292    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442314    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf8670f0-936b-46cb-af2d-f0465743f16d-os-release\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442320    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/367e4815-f156-4241-88f1-b8e84258fd04-host\") pod \"node-ca-ckvrn\" (UID: \"367e4815-f156-4241-88f1-b8e84258fd04\") " pod="openshift-image-registry/node-ca-ckvrn"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442337    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f50b4a82-db6d-409a-8c83-eb229b3e2e13-hosts-file\") pod \"node-resolver-8lrn8\" (UID: \"f50b4a82-db6d-409a-8c83-eb229b3e2e13\") " pod="openshift-dns/node-resolver-8lrn8"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442365    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-sys-fs\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442405    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-socket-dir\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442441    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-sys-fs\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.442729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442445    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29c8b524-a2af-4426-a4c7-dc102e6154ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.443216 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442510    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f50b4a82-db6d-409a-8c83-eb229b3e2e13-hosts-file\") pod \"node-resolver-8lrn8\" (UID: \"f50b4a82-db6d-409a-8c83-eb229b3e2e13\") " pod="openshift-dns/node-resolver-8lrn8"
Jul 27 17:28:40.443216 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442525    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf8670f0-936b-46cb-af2d-f0465743f16d-os-release\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.443216 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442545    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f50b4a82-db6d-409a-8c83-eb229b3e2e13-tmp-dir\") pod \"node-resolver-8lrn8\" (UID: \"f50b4a82-db6d-409a-8c83-eb229b3e2e13\") " pod="openshift-dns/node-resolver-8lrn8"
Jul 27 17:28:40.443216 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442645    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf8670f0-936b-46cb-af2d-f0465743f16d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.443216 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442653    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf8670f0-936b-46cb-af2d-f0465743f16d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.443216 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.442669    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf8670f0-936b-46cb-af2d-f0465743f16d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.451596 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.451564    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:40.451718 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.451706    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:40.451794 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.451786    2571 projected.go:196] Error preparing data for projected volume kube-api-access-rcmlz for pod openshift-network-diagnostics/network-check-target-7qfvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:40.451956 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.451944    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz podName:6bb08eb3-0b94-49f8-8c53-650e6ae67145 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:40.951926067 +0000 UTC m=+2.996664143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rcmlz" (UniqueName: "kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz") pod "network-check-target-7qfvm" (UID: "6bb08eb3-0b94-49f8-8c53-650e6ae67145") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:40.452946 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.452922    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-795z9\" (UniqueName: \"kubernetes.io/projected/cf8670f0-936b-46cb-af2d-f0465743f16d-kube-api-access-795z9\") pod \"multus-additional-cni-plugins-xdrd5\" (UID: \"cf8670f0-936b-46cb-af2d-f0465743f16d\") " pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.453062 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.452926    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl5x8\" (UniqueName: \"kubernetes.io/projected/29c8b524-a2af-4426-a4c7-dc102e6154ac-kube-api-access-pl5x8\") pod \"aws-ebs-csi-driver-node-q4xrx\" (UID: \"29c8b524-a2af-4426-a4c7-dc102e6154ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.453822 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.453788    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g4sl\" (UniqueName: \"kubernetes.io/projected/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-kube-api-access-9g4sl\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:40.453909 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.453827    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5c8l\" (UniqueName: \"kubernetes.io/projected/f50b4a82-db6d-409a-8c83-eb229b3e2e13-kube-api-access-r5c8l\") pod \"node-resolver-8lrn8\" (UID: \"f50b4a82-db6d-409a-8c83-eb229b3e2e13\") " pod="openshift-dns/node-resolver-8lrn8"
Jul 27 17:28:40.454376 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.454353    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2s9\" (UniqueName: \"kubernetes.io/projected/367e4815-f156-4241-88f1-b8e84258fd04-kube-api-access-vx2s9\") pod \"node-ca-ckvrn\" (UID: \"367e4815-f156-4241-88f1-b8e84258fd04\") " pod="openshift-image-registry/node-ca-ckvrn"
Jul 27 17:28:40.530601 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.530548    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k5v8t"
Jul 27 17:28:40.537902 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.537878    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-njw42"
Jul 27 17:28:40.541874 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:40.541849    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bb8c6b_57ef_45fa_bfd9_8c01892ca798.slice/crio-1d8e2ae49069d6b2bdb57b45ddd195950f62b239283a6f4ff90de46afabbdea7 WatchSource:0}: Error finding container 1d8e2ae49069d6b2bdb57b45ddd195950f62b239283a6f4ff90de46afabbdea7: Status 404 returned error can't find the container with id 1d8e2ae49069d6b2bdb57b45ddd195950f62b239283a6f4ff90de46afabbdea7
Jul 27 17:28:40.545970 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.545825    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2vfdv"
Jul 27 17:28:40.547321 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:40.547289    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa19a15_4b11_4bb9_a47a_be6eb5d581db.slice/crio-fce1935643682ec20c997f4e7f478d1a4171010194a8445816f769965726d500 WatchSource:0}: Error finding container fce1935643682ec20c997f4e7f478d1a4171010194a8445816f769965726d500: Status 404 returned error can't find the container with id fce1935643682ec20c997f4e7f478d1a4171010194a8445816f769965726d500
Jul 27 17:28:40.549770 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.549758    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:28:40.555995 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.555946    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cq6fr"
Jul 27 17:28:40.559166 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:40.559141    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0eab36_cd38_40c9_b640_92692454514f.slice/crio-240129c93c3f39095f9b4aabbcf58720ae6ad21fee9eb73286d166b3ccc7e134 WatchSource:0}: Error finding container 240129c93c3f39095f9b4aabbcf58720ae6ad21fee9eb73286d166b3ccc7e134: Status 404 returned error can't find the container with id 240129c93c3f39095f9b4aabbcf58720ae6ad21fee9eb73286d166b3ccc7e134
Jul 27 17:28:40.561583 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.561565    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx"
Jul 27 17:28:40.564032 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:40.563999    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926eec82_2a89_4e47_995c_0de36ce09d69.slice/crio-b326978d87f170ee1de1840e9b7189c722eebfb651bc9c305d90eca9746dad35 WatchSource:0}: Error finding container b326978d87f170ee1de1840e9b7189c722eebfb651bc9c305d90eca9746dad35: Status 404 returned error can't find the container with id b326978d87f170ee1de1840e9b7189c722eebfb651bc9c305d90eca9746dad35
Jul 27 17:28:40.566144 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:40.566119    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a89ad7b_c8a0_411a_86b2_d5f70e7c00fa.slice/crio-573bb8f14b75e50d88fae11d64a25289d3cf9d8f47f5cdb0028f2660e048593f WatchSource:0}: Error finding container 573bb8f14b75e50d88fae11d64a25289d3cf9d8f47f5cdb0028f2660e048593f: Status 404 returned error can't find the container with id 573bb8f14b75e50d88fae11d64a25289d3cf9d8f47f5cdb0028f2660e048593f
Jul 27 17:28:40.566232 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.566180    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ckvrn"
Jul 27 17:28:40.571139 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:40.571117    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c8b524_a2af_4426_a4c7_dc102e6154ac.slice/crio-8d4bd09dec58eebb51084f9ca00c8d84248d143924242244b3a536698106a660 WatchSource:0}: Error finding container 8d4bd09dec58eebb51084f9ca00c8d84248d143924242244b3a536698106a660: Status 404 returned error can't find the container with id 8d4bd09dec58eebb51084f9ca00c8d84248d143924242244b3a536698106a660
Jul 27 17:28:40.571231 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.571129    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xdrd5"
Jul 27 17:28:40.575267 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.575248    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8lrn8"
Jul 27 17:28:40.575387 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:40.575359    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367e4815_f156_4241_88f1_b8e84258fd04.slice/crio-702d7f8adc2004abe33014e69f38705fd4272c702ae810783e3f58375674f74f WatchSource:0}: Error finding container 702d7f8adc2004abe33014e69f38705fd4272c702ae810783e3f58375674f74f: Status 404 returned error can't find the container with id 702d7f8adc2004abe33014e69f38705fd4272c702ae810783e3f58375674f74f
Jul 27 17:28:40.582349 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:40.582327    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8670f0_936b_46cb_af2d_f0465743f16d.slice/crio-308a89ae32d861030e6d8556f9aceb718dd7642d884ea84029bfc778f5886f07 WatchSource:0}: Error finding container 308a89ae32d861030e6d8556f9aceb718dd7642d884ea84029bfc778f5886f07: Status 404 returned error can't find the container with id 308a89ae32d861030e6d8556f9aceb718dd7642d884ea84029bfc778f5886f07
Jul 27 17:28:40.585476 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:40.585449    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50b4a82_db6d_409a_8c83_eb229b3e2e13.slice/crio-6d091e5e08ce353634a372a4e40b40aeb251625b4a3ab219e9b75920a39f0428 WatchSource:0}: Error finding container 6d091e5e08ce353634a372a4e40b40aeb251625b4a3ab219e9b75920a39f0428: Status 404 returned error can't find the container with id 6d091e5e08ce353634a372a4e40b40aeb251625b4a3ab219e9b75920a39f0428
Jul 27 17:28:40.858003 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:28:40.857974    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6587cb3b426ccd7ab481b25ab3a2091.slice/crio-945536406d532380f5c08e2e143b0808b2abbb6c343f2ecfb4fb0c2574b6b6f3 WatchSource:0}: Error finding container 945536406d532380f5c08e2e143b0808b2abbb6c343f2ecfb4fb0c2574b6b6f3: Status 404 returned error can't find the container with id 945536406d532380f5c08e2e143b0808b2abbb6c343f2ecfb4fb0c2574b6b6f3
Jul 27 17:28:40.945606 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.945569    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:40.945766 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:40.945634    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:40.945829 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.945775    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:40.945878 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.945840    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs podName:50fb06f3-ab71-43cc-aa9c-58ba1b7651d6 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:41.945819453 +0000 UTC m=+3.990557515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs") pod "network-metrics-daemon-vtptl" (UID: "50fb06f3-ab71-43cc-aa9c-58ba1b7651d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:40.946273 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.946238    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:40.946435 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:40.946305    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret podName:cbe546c0-bfc4-4692-a40b-3f0d22c0760d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:41.946288049 +0000 UTC m=+3.991026122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret") pod "global-pull-secret-syncer-8kxfj" (UID: "cbe546c0-bfc4-4692-a40b-3f0d22c0760d") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:41.046203 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.046171    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmlz\" (UniqueName: \"kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz\") pod \"network-check-target-7qfvm\" (UID: \"6bb08eb3-0b94-49f8-8c53-650e6ae67145\") " pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:41.046375 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:41.046337    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:41.046375 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:41.046360    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:41.046375 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:41.046370    2571 projected.go:196] Error preparing data for projected volume kube-api-access-rcmlz for pod openshift-network-diagnostics/network-check-target-7qfvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:41.046553 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:41.046418    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz podName:6bb08eb3-0b94-49f8-8c53-650e6ae67145 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:42.046404117 +0000 UTC m=+4.091142192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rcmlz" (UniqueName: "kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz") pod "network-check-target-7qfvm" (UID: "6bb08eb3-0b94-49f8-8c53-650e6ae67145") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:41.271507 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.271325    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 17:23:40 +0000 UTC" deadline="2027-12-29 10:56:13.904130137 +0000 UTC"
Jul 27 17:28:41.271507 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.271388    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12473h27m32.632747903s"
Jul 27 17:28:41.393107 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.393079    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:41.393239 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:41.393216    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:28:41.413322 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.412328    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx" event={"ID":"29c8b524-a2af-4426-a4c7-dc102e6154ac","Type":"ContainerStarted","Data":"8d4bd09dec58eebb51084f9ca00c8d84248d143924242244b3a536698106a660"}
Jul 27 17:28:41.414583 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.414457    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-njw42" event={"ID":"7aa19a15-4b11-4bb9-a47a-be6eb5d581db","Type":"ContainerStarted","Data":"fce1935643682ec20c997f4e7f478d1a4171010194a8445816f769965726d500"}
Jul 27 17:28:41.420828 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.420341    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k5v8t" event={"ID":"55bb8c6b-57ef-45fa-bfd9-8c01892ca798","Type":"ContainerStarted","Data":"1d8e2ae49069d6b2bdb57b45ddd195950f62b239283a6f4ff90de46afabbdea7"}
Jul 27 17:28:41.423400 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.423339    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal" event={"ID":"f6587cb3b426ccd7ab481b25ab3a2091","Type":"ContainerStarted","Data":"945536406d532380f5c08e2e143b0808b2abbb6c343f2ecfb4fb0c2574b6b6f3"}
Jul 27 17:28:41.426097 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.426028    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8lrn8" event={"ID":"f50b4a82-db6d-409a-8c83-eb229b3e2e13","Type":"ContainerStarted","Data":"6d091e5e08ce353634a372a4e40b40aeb251625b4a3ab219e9b75920a39f0428"}
Jul 27 17:28:41.431793 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.431732    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdrd5" event={"ID":"cf8670f0-936b-46cb-af2d-f0465743f16d","Type":"ContainerStarted","Data":"308a89ae32d861030e6d8556f9aceb718dd7642d884ea84029bfc778f5886f07"}
Jul 27 17:28:41.434551 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.434389    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ckvrn" event={"ID":"367e4815-f156-4241-88f1-b8e84258fd04","Type":"ContainerStarted","Data":"702d7f8adc2004abe33014e69f38705fd4272c702ae810783e3f58375674f74f"}
Jul 27 17:28:41.437004 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.436922    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cq6fr" event={"ID":"7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa","Type":"ContainerStarted","Data":"573bb8f14b75e50d88fae11d64a25289d3cf9d8f47f5cdb0028f2660e048593f"}
Jul 27 17:28:41.440361 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.440261    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp" event={"ID":"926eec82-2a89-4e47-995c-0de36ce09d69","Type":"ContainerStarted","Data":"b326978d87f170ee1de1840e9b7189c722eebfb651bc9c305d90eca9746dad35"}
Jul 27 17:28:41.442397 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.442338    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2vfdv" event={"ID":"ae0eab36-cd38-40c9-b640-92692454514f","Type":"ContainerStarted","Data":"240129c93c3f39095f9b4aabbcf58720ae6ad21fee9eb73286d166b3ccc7e134"}
Jul 27 17:28:41.956882 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.956683    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:41.957060 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:41.956920    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:41.957139 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:41.957079    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:41.957192 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:41.957139    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs podName:50fb06f3-ab71-43cc-aa9c-58ba1b7651d6 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:43.957120384 +0000 UTC m=+6.001858463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs") pod "network-metrics-daemon-vtptl" (UID: "50fb06f3-ab71-43cc-aa9c-58ba1b7651d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:41.957566 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:41.957547    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:41.957688 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:41.957605    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret podName:cbe546c0-bfc4-4692-a40b-3f0d22c0760d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:43.957588229 +0000 UTC m=+6.002326302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret") pod "global-pull-secret-syncer-8kxfj" (UID: "cbe546c0-bfc4-4692-a40b-3f0d22c0760d") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:42.058129 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:42.058010    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmlz\" (UniqueName: \"kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz\") pod \"network-check-target-7qfvm\" (UID: \"6bb08eb3-0b94-49f8-8c53-650e6ae67145\") " pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:42.058254 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:42.058185    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:42.058254 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:42.058204    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:42.058254 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:42.058215    2571 projected.go:196] Error preparing data for projected volume kube-api-access-rcmlz for pod openshift-network-diagnostics/network-check-target-7qfvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:42.058418 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:42.058303    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz podName:6bb08eb3-0b94-49f8-8c53-650e6ae67145 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:44.058270912 +0000 UTC m=+6.103008972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rcmlz" (UniqueName: "kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz") pod "network-check-target-7qfvm" (UID: "6bb08eb3-0b94-49f8-8c53-650e6ae67145") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:42.272352 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:42.272201    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 17:23:40 +0000 UTC" deadline="2028-03-12 06:16:26.864333313 +0000 UTC"
Jul 27 17:28:42.272352 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:42.272241    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14244h47m44.592096251s"
Jul 27 17:28:42.398512 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:42.397807    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:42.398512 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:42.397931    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:28:42.398512 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:42.398341    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:42.398512 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:42.398433    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:28:42.470110 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:42.469260    2571 generic.go:356] "Generic (PLEG): container finished" podID="ea5219289a853fa387fd5432a83f50ac" containerID="5bb60b202024bb4a3612bc4e07745dcf6199bde015fe72a93962cdf8229bbd38" exitCode=0
Jul 27 17:28:42.470110 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:42.469307    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal" event={"ID":"ea5219289a853fa387fd5432a83f50ac","Type":"ContainerDied","Data":"5bb60b202024bb4a3612bc4e07745dcf6199bde015fe72a93962cdf8229bbd38"}
Jul 27 17:28:43.393467 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:43.392917    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:43.393467 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:43.393061    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:28:43.490289 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:43.490099    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal" event={"ID":"ea5219289a853fa387fd5432a83f50ac","Type":"ContainerStarted","Data":"3184538c8579db94e2df50844ef7092c7b6d80492da7da23805da9af2ae2a75e"}
Jul 27 17:28:43.974300 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:43.974262    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:43.974489 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:43.974325    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:43.974489 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:43.974462    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:43.974685 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:43.974539    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs podName:50fb06f3-ab71-43cc-aa9c-58ba1b7651d6 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:47.974520011 +0000 UTC m=+10.019258084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs") pod "network-metrics-daemon-vtptl" (UID: "50fb06f3-ab71-43cc-aa9c-58ba1b7651d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:43.974955 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:43.974937    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:43.975024 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:43.974991    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret podName:cbe546c0-bfc4-4692-a40b-3f0d22c0760d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:47.974975881 +0000 UTC m=+10.019713954 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret") pod "global-pull-secret-syncer-8kxfj" (UID: "cbe546c0-bfc4-4692-a40b-3f0d22c0760d") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:44.075344 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:44.075299    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmlz\" (UniqueName: \"kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz\") pod \"network-check-target-7qfvm\" (UID: \"6bb08eb3-0b94-49f8-8c53-650e6ae67145\") " pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:44.075531 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:44.075466    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:44.075531 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:44.075486    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:44.075531 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:44.075513    2571 projected.go:196] Error preparing data for projected volume kube-api-access-rcmlz for pod openshift-network-diagnostics/network-check-target-7qfvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:44.075720 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:44.075568    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz podName:6bb08eb3-0b94-49f8-8c53-650e6ae67145 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:48.075550458 +0000 UTC m=+10.120288531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rcmlz" (UniqueName: "kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz") pod "network-check-target-7qfvm" (UID: "6bb08eb3-0b94-49f8-8c53-650e6ae67145") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:44.394207 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:44.393581    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:44.394207 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:44.393713    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:28:44.394207 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:44.393581    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:44.394207 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:44.394125    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:28:45.394018 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:45.393524    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:45.394018 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:45.393665    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:28:46.393506 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:46.393461    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:46.393953 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:46.393612    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:28:46.393953 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:46.393651    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:46.393953 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:46.393756    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:28:47.393350 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:47.393314    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:47.393539 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:47.393480    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:28:48.006663 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:48.005788    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:48.006663 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:48.005915    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:48.006663 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:48.006095    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:48.006663 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:48.006163    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs podName:50fb06f3-ab71-43cc-aa9c-58ba1b7651d6 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:56.006143676 +0000 UTC m=+18.050881755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs") pod "network-metrics-daemon-vtptl" (UID: "50fb06f3-ab71-43cc-aa9c-58ba1b7651d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:48.006663 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:48.006588    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:48.006663 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:48.006637    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret podName:cbe546c0-bfc4-4692-a40b-3f0d22c0760d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:56.006622068 +0000 UTC m=+18.051360141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret") pod "global-pull-secret-syncer-8kxfj" (UID: "cbe546c0-bfc4-4692-a40b-3f0d22c0760d") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:48.107253 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:48.107210    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmlz\" (UniqueName: \"kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz\") pod \"network-check-target-7qfvm\" (UID: \"6bb08eb3-0b94-49f8-8c53-650e6ae67145\") " pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:48.107414 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:48.107375    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:48.107414 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:48.107403    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:48.107414 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:48.107416    2571 projected.go:196] Error preparing data for projected volume kube-api-access-rcmlz for pod openshift-network-diagnostics/network-check-target-7qfvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:48.107543 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:48.107483    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz podName:6bb08eb3-0b94-49f8-8c53-650e6ae67145 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:56.107464229 +0000 UTC m=+18.152202309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rcmlz" (UniqueName: "kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz") pod "network-check-target-7qfvm" (UID: "6bb08eb3-0b94-49f8-8c53-650e6ae67145") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:48.394825 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:48.394243    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:48.394825 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:48.394369    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:28:48.394825 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:48.394409    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:48.394825 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:48.394470    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:28:49.393399 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:49.393367    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:49.393607 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:49.393510    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:28:50.393999 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:50.393967    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:50.393999 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:50.393988    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:50.394555 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:50.394116    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:28:50.394751 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:50.394725    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:28:51.393873 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:51.393838    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:51.394042 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:51.393964    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:28:52.396716 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:52.396677    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:52.397158 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:52.396677    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:52.397158 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:52.396812    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:28:52.397158 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:52.396903    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:28:53.393685 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:53.393652    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:53.393855 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:53.393785    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:28:54.397314 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:54.397281    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:54.397727 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:54.397405    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:28:54.397727 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:54.397454    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:54.397727 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:54.397586    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:28:55.393254 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:55.393221    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:55.393454 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:55.393371    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:28:56.066430 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:56.066391    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:56.066937 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:56.066445    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:56.066937 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:56.066537    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:56.066937 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:56.066551    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:56.066937 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:56.066606    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret podName:cbe546c0-bfc4-4692-a40b-3f0d22c0760d nodeName:}" failed. No retries permitted until 2026-07-27 17:29:12.06658624 +0000 UTC m=+34.111324300 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret") pod "global-pull-secret-syncer-8kxfj" (UID: "cbe546c0-bfc4-4692-a40b-3f0d22c0760d") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:56.066937 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:56.066626    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs podName:50fb06f3-ab71-43cc-aa9c-58ba1b7651d6 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:12.066618644 +0000 UTC m=+34.111356703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs") pod "network-metrics-daemon-vtptl" (UID: "50fb06f3-ab71-43cc-aa9c-58ba1b7651d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:56.167384 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:56.167355    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmlz\" (UniqueName: \"kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz\") pod \"network-check-target-7qfvm\" (UID: \"6bb08eb3-0b94-49f8-8c53-650e6ae67145\") " pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:56.167573 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:56.167484    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:56.167573 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:56.167514    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:56.167573 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:56.167525    2571 projected.go:196] Error preparing data for projected volume kube-api-access-rcmlz for pod openshift-network-diagnostics/network-check-target-7qfvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:56.167702 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:56.167576    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz podName:6bb08eb3-0b94-49f8-8c53-650e6ae67145 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:12.167560153 +0000 UTC m=+34.212298231 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rcmlz" (UniqueName: "kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz") pod "network-check-target-7qfvm" (UID: "6bb08eb3-0b94-49f8-8c53-650e6ae67145") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:56.393735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:56.393658    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:56.393946 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:56.393658    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:56.393946 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:56.393778    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:28:56.393946 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:56.393886    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:28:57.393761 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:57.393722    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:57.394192 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:57.393854    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:28:58.394989 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:58.394947    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:28:58.395440 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:58.395058    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:28:58.395440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:58.395111    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:28:58.395440 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:58.395251    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:28:59.393775 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:28:59.393737    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:28:59.393961 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:28:59.393855    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:29:00.393308 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.393147    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:29:00.393308 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.393149    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:29:00.393308 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:00.393267    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:29:00.393991 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:00.393393    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:29:00.526742 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.526577    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx" event={"ID":"29c8b524-a2af-4426-a4c7-dc102e6154ac","Type":"ContainerStarted","Data":"1bd1dbcd34e5006c8a6c5b7bbba2da2d24bf02743d71f3aa07933aea0b066968"}
Jul 27 17:29:00.528091 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.528063    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-njw42" event={"ID":"7aa19a15-4b11-4bb9-a47a-be6eb5d581db","Type":"ContainerStarted","Data":"511da68e34a7fae6c2e88542a46d82389a8db739542a45e77a11b697ea67307c"}
Jul 27 17:29:00.529864 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.529843    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k5v8t" event={"ID":"55bb8c6b-57ef-45fa-bfd9-8c01892ca798","Type":"ContainerStarted","Data":"24b95cc40f7ed00e4cad0f677357b63ee4e4473d36a394f65fadab73930f7742"}
Jul 27 17:29:00.531263 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.531238    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal" event={"ID":"f6587cb3b426ccd7ab481b25ab3a2091","Type":"ContainerStarted","Data":"53ca7af4658d0a8472aa92cb462fa0990b50ef6cf7e5e04a19138c930dfe42d0"}
Jul 27 17:29:00.532641 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.532620    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8lrn8" event={"ID":"f50b4a82-db6d-409a-8c83-eb229b3e2e13","Type":"ContainerStarted","Data":"a0929a1f1d6f3964507b3934a5b26d0e26ef20c01ea33c9d5d693675c330b7b4"}
Jul 27 17:29:00.534070 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.534044    2571 generic.go:356] "Generic (PLEG): container finished" podID="cf8670f0-936b-46cb-af2d-f0465743f16d" containerID="972f6d235b14f46f7a4aa8fa550af811229d66bcc2a7aed600fe68807e7306a9" exitCode=0
Jul 27 17:29:00.534164 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.534121    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdrd5" event={"ID":"cf8670f0-936b-46cb-af2d-f0465743f16d","Type":"ContainerDied","Data":"972f6d235b14f46f7a4aa8fa550af811229d66bcc2a7aed600fe68807e7306a9"}
Jul 27 17:29:00.535667 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.535604    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ckvrn" event={"ID":"367e4815-f156-4241-88f1-b8e84258fd04","Type":"ContainerStarted","Data":"af2485fe53607f01b4a2b8987d1750a3e047199ab58c272ac85fd0067e34204d"}
Jul 27 17:29:00.537033 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.537008    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cq6fr" event={"ID":"7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa","Type":"ContainerStarted","Data":"d80480fbb1959ffa7db7af319f8d9ceb6d1d0621f67ab714d91c2ac5b17c8336"}
Jul 27 17:29:00.538679 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.538661    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp" event={"ID":"926eec82-2a89-4e47-995c-0de36ce09d69","Type":"ContainerStarted","Data":"f98aa0eb5f3680b6dd2c03d2e5d8d6439c877b58da720ab15099399f638dc6b2"}
Jul 27 17:29:00.538782 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.538731    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp" event={"ID":"926eec82-2a89-4e47-995c-0de36ce09d69","Type":"ContainerStarted","Data":"fe0c06d127a9ef2bcdf08e1b34ea1cda788e8c9dc87f5b28dc0f06bc8ff94b94"}
Jul 27 17:29:00.545116 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.545072    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-74.ec2.internal" podStartSLOduration=21.545061408 podStartE2EDuration="21.545061408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:43.50699045 +0000 UTC m=+5.551728533" watchObservedRunningTime="2026-07-27 17:29:00.545061408 +0000 UTC m=+22.589799489"
Jul 27 17:29:00.545242 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.545205    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-njw42" podStartSLOduration=2.960623892 podStartE2EDuration="22.545199872s" totalImagesPullingTime="19.58457598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:00.544785153 +0000 UTC m=+22.589523235" watchObservedRunningTime="2026-07-27 17:29:00.545199872 +0000 UTC m=+22.589937974"
Jul 27 17:29:00.558853 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.558818    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ckvrn" podStartSLOduration=7.213955482 podStartE2EDuration="22.558806426s" totalImagesPullingTime="15.344850944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:00.558669005 +0000 UTC m=+22.603407086" watchObservedRunningTime="2026-07-27 17:29:00.558806426 +0000 UTC m=+22.603544507"
Jul 27 17:29:00.594665 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.594630    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k5v8t" podStartSLOduration=3.006437198 podStartE2EDuration="22.594615734s" totalImagesPullingTime="19.588178536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:00.594328007 +0000 UTC m=+22.639066088" watchObservedRunningTime="2026-07-27 17:29:00.594615734 +0000 UTC m=+22.639353814"
Jul 27 17:29:00.610074 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.610036    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8lrn8" podStartSLOduration=3.125956204 podStartE2EDuration="22.610022914s" totalImagesPullingTime="19.48406671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:00.609729255 +0000 UTC m=+22.654467336" watchObservedRunningTime="2026-07-27 17:29:00.610022914 +0000 UTC m=+22.654760995"
Jul 27 17:29:00.624317 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.624272    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal" podStartSLOduration=21.624254605 podStartE2EDuration="21.624254605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:00.624093545 +0000 UTC m=+22.668831651" watchObservedRunningTime="2026-07-27 17:29:00.624254605 +0000 UTC m=+22.668992687"
Jul 27 17:29:00.639643 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:00.639609    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cq6fr" podStartSLOduration=11.029721279 podStartE2EDuration="22.639596552s" totalImagesPullingTime="11.609875273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:00.639486459 +0000 UTC m=+22.684224539" watchObservedRunningTime="2026-07-27 17:29:00.639596552 +0000 UTC m=+22.684334632"
Jul 27 17:29:01.393142 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:01.393110    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:29:01.393271 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:01.393207    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:29:01.543830 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:01.543798    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp" event={"ID":"926eec82-2a89-4e47-995c-0de36ce09d69","Type":"ContainerStarted","Data":"13c9b8d0e7498776dfce5bf94b046702c1e5517fa5b08deea50ae61051446ed5"}
Jul 27 17:29:01.544211 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:01.543895    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp" event={"ID":"926eec82-2a89-4e47-995c-0de36ce09d69","Type":"ContainerStarted","Data":"0ca8baaa91616cd9e2d206ad14b8d6a14c0d42705e50f7fa2d58c3d290c914b2"}
Jul 27 17:29:01.544211 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:01.543959    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp" event={"ID":"926eec82-2a89-4e47-995c-0de36ce09d69","Type":"ContainerStarted","Data":"612c4aca122b00b02d2b02c08d97b5fb7a25c489155dc20db52cb157adf2757a"}
Jul 27 17:29:01.544211 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:01.544006    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp" event={"ID":"926eec82-2a89-4e47-995c-0de36ce09d69","Type":"ContainerStarted","Data":"fc17c7c65f86277aeaec4922b1240883a023e1ce8a51f40ebadab37a68af6d0d"}
Jul 27 17:29:01.545849 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:01.545783    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2vfdv" event={"ID":"ae0eab36-cd38-40c9-b640-92692454514f","Type":"ContainerStarted","Data":"ec76c9b551ddd54cd63c1d5517decd686c41a95d9363e821406eb19ed0687d46"}
Jul 27 17:29:01.561380 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:01.561333    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2vfdv" podStartSLOduration=4.051658415 podStartE2EDuration="23.561316645s" totalImagesPullingTime="19.50965823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:01.561056403 +0000 UTC m=+23.605794484" watchObservedRunningTime="2026-07-27 17:29:01.561316645 +0000 UTC m=+23.606054727"
Jul 27 17:29:01.624846 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:01.624822    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 17:29:02.306580 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:02.306450    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T17:29:01.624843144Z","UUID":"fdbedd6a-e749-4d17-9f9d-ea57f00f3703","Handler":null,"Name":"","Endpoint":""}
Jul 27 17:29:02.308239 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:02.308218    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 17:29:02.308379 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:02.308259    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 17:29:02.393975 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:02.393946    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:29:02.394161 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:02.393951    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:29:02.394161 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:02.394082    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:29:02.394274 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:02.394162    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:29:02.543713 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:02.543679    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq6fr_7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:02.550930 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:02.550894    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx" event={"ID":"29c8b524-a2af-4426-a4c7-dc102e6154ac","Type":"ContainerStarted","Data":"b66a9976c89e9a16eeb110aeb1a9a1ea14b24a12595b2bd689df0832fd2acd36"}
Jul 27 17:29:02.639424 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:02.639351    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq6fr_7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:03.298124 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:03.298037    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cq6fr"
Jul 27 17:29:03.298684 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:03.298666    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cq6fr"
Jul 27 17:29:03.393889 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:03.393865    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:29:03.394095 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:03.393977    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:29:03.557304 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:03.557156    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp" event={"ID":"926eec82-2a89-4e47-995c-0de36ce09d69","Type":"ContainerStarted","Data":"c59bb8508dafec189bc78f34c98c3fcffa8004e0e9c3323cfa033050701f6ab1"}
Jul 27 17:29:03.559575 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:03.559546    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx" event={"ID":"29c8b524-a2af-4426-a4c7-dc102e6154ac","Type":"ContainerStarted","Data":"700d0e25b12a9fff5af7d3a7851f7a914a1279e3599f0069e3b596137831db29"}
Jul 27 17:29:03.560080 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:03.559819    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cq6fr"
Jul 27 17:29:03.560319 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:03.560301    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cq6fr"
Jul 27 17:29:03.579665 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:03.579609    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4xrx" podStartSLOduration=11.278518655 podStartE2EDuration="25.579591782s" totalImagesPullingTime="14.301073127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:03.579431648 +0000 UTC m=+25.624169729" watchObservedRunningTime="2026-07-27 17:29:03.579591782 +0000 UTC m=+25.624329864"
Jul 27 17:29:03.641022 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:03.640989    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq6fr_7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:04.393687 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:04.393661    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:29:04.393872 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:04.393790    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:29:04.393872 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:04.393809    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:29:04.393990 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:04.393939    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:29:05.393539 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:05.393441    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:29:05.394124 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:05.393612    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:29:06.394053 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:06.393845    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:29:06.394538 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:06.393905    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:29:06.394538 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:06.394089    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:29:06.394538 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:06.394187    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:29:06.444883 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:06.444850    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq6fr_7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:06.498788 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:06.498749    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq6fr_7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:06.567917 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:06.567881    2571 generic.go:356] "Generic (PLEG): container finished" podID="cf8670f0-936b-46cb-af2d-f0465743f16d" containerID="f4d77fd8727cfba2223d459af917e925b7a7987d87790cb78cd52008896b4112" exitCode=0
Jul 27 17:29:06.568035 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:06.567971    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdrd5" event={"ID":"cf8670f0-936b-46cb-af2d-f0465743f16d","Type":"ContainerDied","Data":"f4d77fd8727cfba2223d459af917e925b7a7987d87790cb78cd52008896b4112"}
Jul 27 17:29:06.571560 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:06.571542    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp" event={"ID":"926eec82-2a89-4e47-995c-0de36ce09d69","Type":"ContainerStarted","Data":"8d2076f1615c8e9c49720fc61e9e8c146b4a5571b77027ff4aee029e7c14e491"}
Jul 27 17:29:06.574179 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:06.573280    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:29:06.574179 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:06.573417    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:29:06.589547 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:06.589524    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:29:06.593921 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:06.593899    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:29:06.615537 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:06.615482    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp" podStartSLOduration=8.922895562 podStartE2EDuration="28.615470191s" totalImagesPullingTime="19.692574629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:06.614097397 +0000 UTC m=+28.658835491" watchObservedRunningTime="2026-07-27 17:29:06.615470191 +0000 UTC m=+28.660208284"
Jul 27 17:29:07.035385 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:07.035353    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq6fr_7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:07.393187 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:07.393125    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:29:07.393325 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:07.393241    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:29:07.576007 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:07.575981    2571 generic.go:356] "Generic (PLEG): container finished" podID="cf8670f0-936b-46cb-af2d-f0465743f16d" containerID="cbc9ad4e83a80dabad3324cebb840f7b7ca8b017344ae3ef64e61a6cc9eaf984" exitCode=0
Jul 27 17:29:07.576429 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:07.576054    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdrd5" event={"ID":"cf8670f0-936b-46cb-af2d-f0465743f16d","Type":"ContainerDied","Data":"cbc9ad4e83a80dabad3324cebb840f7b7ca8b017344ae3ef64e61a6cc9eaf984"}
Jul 27 17:29:07.576429 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:07.576270    2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 17:29:07.860450 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:07.860178    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8kxfj"]
Jul 27 17:29:07.860669 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:07.860605    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:29:07.860748 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:07.860717    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:29:07.860919 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:07.860897    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7qfvm"]
Jul 27 17:29:07.861018 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:07.860992    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:29:07.861128 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:07.861068    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:29:07.863617 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:07.863597    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vtptl"]
Jul 27 17:29:07.863713 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:07.863688    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:29:07.863786 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:07.863769    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:29:08.578260 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:08.578065    2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 17:29:08.717303 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:08.717280    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:29:09.393231 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:09.393198    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:29:09.393231 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:09.393233    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:29:09.393425 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:09.393211    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:29:09.393425 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:09.393320    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:29:09.393425 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:09.393364    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:29:09.393567 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:09.393430    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:29:09.585508 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:09.585411    2571 generic.go:356] "Generic (PLEG): container finished" podID="cf8670f0-936b-46cb-af2d-f0465743f16d" containerID="6532b7fbad3a66038d3c6f2c0bb573e8a07f7aca4339cf46955c40079fb63a79" exitCode=0
Jul 27 17:29:09.585917 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:09.585491    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdrd5" event={"ID":"cf8670f0-936b-46cb-af2d-f0465743f16d","Type":"ContainerDied","Data":"6532b7fbad3a66038d3c6f2c0bb573e8a07f7aca4339cf46955c40079fb63a79"}
Jul 27 17:29:11.393316 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:11.393285    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:29:11.393867 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:11.393332    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:29:11.393867 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:11.393333    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:29:11.393867 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:11.393405    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qfvm" podUID="6bb08eb3-0b94-49f8-8c53-650e6ae67145"
Jul 27 17:29:11.393867 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:11.393512    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8kxfj" podUID="cbe546c0-bfc4-4692-a40b-3f0d22c0760d"
Jul 27 17:29:11.393867 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:11.393638    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:29:12.083636 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.083594    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:29:12.083832 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.083661    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:29:12.083832 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:12.083729    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:29:12.083832 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:12.083806    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret podName:cbe546c0-bfc4-4692-a40b-3f0d22c0760d nodeName:}" failed. No retries permitted until 2026-07-27 17:29:44.083788718 +0000 UTC m=+66.128526781 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret") pod "global-pull-secret-syncer-8kxfj" (UID: "cbe546c0-bfc4-4692-a40b-3f0d22c0760d") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:29:12.083832 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:12.083819    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:29:12.084041 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:12.083894    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs podName:50fb06f3-ab71-43cc-aa9c-58ba1b7651d6 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:44.083876265 +0000 UTC m=+66.128614333 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs") pod "network-metrics-daemon-vtptl" (UID: "50fb06f3-ab71-43cc-aa9c-58ba1b7651d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:29:12.184405 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.184368    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmlz\" (UniqueName: \"kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz\") pod \"network-check-target-7qfvm\" (UID: \"6bb08eb3-0b94-49f8-8c53-650e6ae67145\") " pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:29:12.184601 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:12.184529    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:29:12.184601 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:12.184545    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:29:12.184601 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:12.184554    2571 projected.go:196] Error preparing data for projected volume kube-api-access-rcmlz for pod openshift-network-diagnostics/network-check-target-7qfvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:29:12.184601 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:12.184602    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz podName:6bb08eb3-0b94-49f8-8c53-650e6ae67145 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:44.184590085 +0000 UTC m=+66.229328143 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rcmlz" (UniqueName: "kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz") pod "network-check-target-7qfvm" (UID: "6bb08eb3-0b94-49f8-8c53-650e6ae67145") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:29:12.807164 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.807126    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-74.ec2.internal" event="NodeReady"
Jul 27 17:29:12.807632 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.807392    2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 17:29:12.841510 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.841459    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"]
Jul 27 17:29:12.847337 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.847314    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:12.849897 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.849874    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 17:29:12.849983 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.849907    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 17:29:12.850120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.850106    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 17:29:12.850216 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.850197    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-skq5m\""
Jul 27 17:29:12.855093 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.855070    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zfdj9"]
Jul 27 17:29:12.856607 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.856532    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 17:29:12.859472 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.859451    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dvrgq"]
Jul 27 17:29:12.859615 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.859593    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:12.862642 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.862325    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 17:29:12.862642 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.862414    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x7wp7\""
Jul 27 17:29:12.862642 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.862578    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 17:29:12.864014 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.863994    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"]
Jul 27 17:29:12.864216 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.864188    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dvrgq"
Jul 27 17:29:12.865084 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.864993    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dvrgq"]
Jul 27 17:29:12.866933 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.866786    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 17:29:12.866933 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.866801    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6fr89\""
Jul 27 17:29:12.866933 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.866811    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 17:29:12.867140 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.867108    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 17:29:12.868781 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.868734    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zfdj9"]
Jul 27 17:29:12.977179 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.977081    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq6fr_7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:12.991873 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.991845    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:12.992016 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.991879    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6825326-7c3b-4465-bc99-8f541ccad67f-cert\") pod \"ingress-canary-dvrgq\" (UID: \"f6825326-7c3b-4465-bc99-8f541ccad67f\") " pod="openshift-ingress-canary/ingress-canary-dvrgq"
Jul 27 17:29:12.992016 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.991911    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phw9k\" (UniqueName: \"kubernetes.io/projected/b4177ef3-39b7-4328-9299-d1acac07b82d-kube-api-access-phw9k\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:12.992106 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.992009    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-certificates\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:12.992106 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.992042    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-bound-sa-token\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:12.992106 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.992084    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfp29\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-kube-api-access-vfp29\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:12.992261 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.992140    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4177ef3-39b7-4328-9299-d1acac07b82d-config-volume\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:12.992261 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.992175    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4177ef3-39b7-4328-9299-d1acac07b82d-tmp-dir\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:12.992261 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.992208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5ce3636-8fba-4864-99f7-28f1edd98f4e-installation-pull-secrets\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:12.992261 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.992254    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7twq\" (UniqueName: \"kubernetes.io/projected/f6825326-7c3b-4465-bc99-8f541ccad67f-kube-api-access-q7twq\") pod \"ingress-canary-dvrgq\" (UID: \"f6825326-7c3b-4465-bc99-8f541ccad67f\") " pod="openshift-ingress-canary/ingress-canary-dvrgq"
Jul 27 17:29:12.992408 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.992296    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:12.992408 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.992317    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5ce3636-8fba-4864-99f7-28f1edd98f4e-ca-trust-extracted\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:12.992408 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.992332    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5ce3636-8fba-4864-99f7-28f1edd98f4e-trusted-ca\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:12.992408 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:12.992358    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5ce3636-8fba-4864-99f7-28f1edd98f4e-image-registry-private-configuration\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.093214 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093176    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4177ef3-39b7-4328-9299-d1acac07b82d-config-volume\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:13.093214 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093216    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4177ef3-39b7-4328-9299-d1acac07b82d-tmp-dir\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:13.093489 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093237    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5ce3636-8fba-4864-99f7-28f1edd98f4e-installation-pull-secrets\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.093489 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093255    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7twq\" (UniqueName: \"kubernetes.io/projected/f6825326-7c3b-4465-bc99-8f541ccad67f-kube-api-access-q7twq\") pod \"ingress-canary-dvrgq\" (UID: \"f6825326-7c3b-4465-bc99-8f541ccad67f\") " pod="openshift-ingress-canary/ingress-canary-dvrgq"
Jul 27 17:29:13.093489 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093282    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.093489 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093303    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5ce3636-8fba-4864-99f7-28f1edd98f4e-ca-trust-extracted\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.093489 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:13.093402    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:13.093489 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:13.093422    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ddd5cfc57-4mhc2: secret "image-registry-tls" not found
Jul 27 17:29:13.093489 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093466    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5ce3636-8fba-4864-99f7-28f1edd98f4e-trusted-ca\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.093489 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:13.093487    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls podName:b5ce3636-8fba-4864-99f7-28f1edd98f4e nodeName:}" failed. No retries permitted until 2026-07-27 17:29:13.593468861 +0000 UTC m=+35.638206934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls") pod "image-registry-5ddd5cfc57-4mhc2" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e") : secret "image-registry-tls" not found
Jul 27 17:29:13.093922 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093544    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5ce3636-8fba-4864-99f7-28f1edd98f4e-image-registry-private-configuration\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.093922 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093583    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:13.093922 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093604    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6825326-7c3b-4465-bc99-8f541ccad67f-cert\") pod \"ingress-canary-dvrgq\" (UID: \"f6825326-7c3b-4465-bc99-8f541ccad67f\") " pod="openshift-ingress-canary/ingress-canary-dvrgq"
Jul 27 17:29:13.093922 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093630    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phw9k\" (UniqueName: \"kubernetes.io/projected/b4177ef3-39b7-4328-9299-d1acac07b82d-kube-api-access-phw9k\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:13.093922 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093670    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4177ef3-39b7-4328-9299-d1acac07b82d-tmp-dir\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:13.093922 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093714    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5ce3636-8fba-4864-99f7-28f1edd98f4e-ca-trust-extracted\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.093922 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:13.093739    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:13.093922 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:13.093788    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls podName:b4177ef3-39b7-4328-9299-d1acac07b82d nodeName:}" failed. No retries permitted until 2026-07-27 17:29:13.593771296 +0000 UTC m=+35.638509387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls") pod "dns-default-zfdj9" (UID: "b4177ef3-39b7-4328-9299-d1acac07b82d") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:13.093922 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093817    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-certificates\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.093922 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093847    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-bound-sa-token\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.093922 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093855    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4177ef3-39b7-4328-9299-d1acac07b82d-config-volume\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:13.093922 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.093888    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfp29\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-kube-api-access-vfp29\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.094641 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.094618    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-certificates\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.094731 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.094679    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5ce3636-8fba-4864-99f7-28f1edd98f4e-trusted-ca\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.098040 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.097905    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5ce3636-8fba-4864-99f7-28f1edd98f4e-installation-pull-secrets\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.098157 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.097925    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5ce3636-8fba-4864-99f7-28f1edd98f4e-image-registry-private-configuration\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.098157 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.097952    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6825326-7c3b-4465-bc99-8f541ccad67f-cert\") pod \"ingress-canary-dvrgq\" (UID: \"f6825326-7c3b-4465-bc99-8f541ccad67f\") " pod="openshift-ingress-canary/ingress-canary-dvrgq"
Jul 27 17:29:13.104881 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.104645    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-bound-sa-token\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.104881 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.104698    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phw9k\" (UniqueName: \"kubernetes.io/projected/b4177ef3-39b7-4328-9299-d1acac07b82d-kube-api-access-phw9k\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:13.104881 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.104841    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfp29\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-kube-api-access-vfp29\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.105193 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.105168    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7twq\" (UniqueName: \"kubernetes.io/projected/f6825326-7c3b-4465-bc99-8f541ccad67f-kube-api-access-q7twq\") pod \"ingress-canary-dvrgq\" (UID: \"f6825326-7c3b-4465-bc99-8f541ccad67f\") " pod="openshift-ingress-canary/ingress-canary-dvrgq"
Jul 27 17:29:13.177423 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.177391    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dvrgq"
Jul 27 17:29:13.393186 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.393070    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:29:13.393344 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.393201    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:29:13.393344 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.393229    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:29:13.396172 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.396147    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 17:29:13.396317 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.396256    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qwvcb\""
Jul 27 17:29:13.396317 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.396311    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 17:29:13.396488 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.396349    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zkh28\""
Jul 27 17:29:13.396619 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.396604    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 17:29:13.397299 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.397283    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 17:29:13.597053 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.597010    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:13.597201 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.597077    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:13.597201 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:13.597182    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:13.597201 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:13.597191    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:13.597201 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:13.597202    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ddd5cfc57-4mhc2: secret "image-registry-tls" not found
Jul 27 17:29:13.597357 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:13.597257    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls podName:b4177ef3-39b7-4328-9299-d1acac07b82d nodeName:}" failed. No retries permitted until 2026-07-27 17:29:14.597239393 +0000 UTC m=+36.641977467 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls") pod "dns-default-zfdj9" (UID: "b4177ef3-39b7-4328-9299-d1acac07b82d") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:13.597357 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:13.597276    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls podName:b5ce3636-8fba-4864-99f7-28f1edd98f4e nodeName:}" failed. No retries permitted until 2026-07-27 17:29:14.59726612 +0000 UTC m=+36.642004179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls") pod "image-registry-5ddd5cfc57-4mhc2" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e") : secret "image-registry-tls" not found
Jul 27 17:29:13.620236 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.620202    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dvrgq"]
Jul 27 17:29:13.623699 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:29:13.623676    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6825326_7c3b_4465_bc99_8f541ccad67f.slice/crio-27c8e46f93da20d5e309f18a3fa18b8da977a070c5eedb5344eeaeb65c9fa97e WatchSource:0}: Error finding container 27c8e46f93da20d5e309f18a3fa18b8da977a070c5eedb5344eeaeb65c9fa97e: Status 404 returned error can't find the container with id 27c8e46f93da20d5e309f18a3fa18b8da977a070c5eedb5344eeaeb65c9fa97e
Jul 27 17:29:13.838390 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:13.838357    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq6fr_7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:14.025095 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:14.025070    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq6fr_7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:14.600009 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:14.599982    2571 generic.go:356] "Generic (PLEG): container finished" podID="cf8670f0-936b-46cb-af2d-f0465743f16d" containerID="74364328459ac17da86afd787c314cafe24f681e8ac049bd9d16277f15d5a2a5" exitCode=0
Jul 27 17:29:14.600185 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:14.600075    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdrd5" event={"ID":"cf8670f0-936b-46cb-af2d-f0465743f16d","Type":"ContainerDied","Data":"74364328459ac17da86afd787c314cafe24f681e8ac049bd9d16277f15d5a2a5"}
Jul 27 17:29:14.601163 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:14.601143    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dvrgq" event={"ID":"f6825326-7c3b-4465-bc99-8f541ccad67f","Type":"ContainerStarted","Data":"27c8e46f93da20d5e309f18a3fa18b8da977a070c5eedb5344eeaeb65c9fa97e"}
Jul 27 17:29:14.603906 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:14.603882    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:14.604140 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:14.603931    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:14.604140 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:14.603966    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:14.604140 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:14.603979    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ddd5cfc57-4mhc2: secret "image-registry-tls" not found
Jul 27 17:29:14.604140 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:14.604022    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls podName:b5ce3636-8fba-4864-99f7-28f1edd98f4e nodeName:}" failed. No retries permitted until 2026-07-27 17:29:16.604009274 +0000 UTC m=+38.648747346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls") pod "image-registry-5ddd5cfc57-4mhc2" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e") : secret "image-registry-tls" not found
Jul 27 17:29:14.604140 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:14.604037    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:14.604140 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:14.604081    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls podName:b4177ef3-39b7-4328-9299-d1acac07b82d nodeName:}" failed. No retries permitted until 2026-07-27 17:29:16.604067197 +0000 UTC m=+38.648805272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls") pod "dns-default-zfdj9" (UID: "b4177ef3-39b7-4328-9299-d1acac07b82d") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:15.605810 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:15.605772    2571 generic.go:356] "Generic (PLEG): container finished" podID="cf8670f0-936b-46cb-af2d-f0465743f16d" containerID="6f1d8065397447abe8ffc3efd5148129506f4e30ee025ef301388caee1220de4" exitCode=0
Jul 27 17:29:15.606164 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:15.605823    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdrd5" event={"ID":"cf8670f0-936b-46cb-af2d-f0465743f16d","Type":"ContainerDied","Data":"6f1d8065397447abe8ffc3efd5148129506f4e30ee025ef301388caee1220de4"}
Jul 27 17:29:16.613018 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:16.612984    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdrd5" event={"ID":"cf8670f0-936b-46cb-af2d-f0465743f16d","Type":"ContainerStarted","Data":"a46a2b9b11f985b3f241aa9ef522a759986ffc065e7882aab2376bb58f99048b"}
Jul 27 17:29:16.617570 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:16.617547    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:16.617684 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:16.617657    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:16.617753 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:16.617698    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:16.617808 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:16.617759    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls podName:b4177ef3-39b7-4328-9299-d1acac07b82d nodeName:}" failed. No retries permitted until 2026-07-27 17:29:20.617738835 +0000 UTC m=+42.662476904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls") pod "dns-default-zfdj9" (UID: "b4177ef3-39b7-4328-9299-d1acac07b82d") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:16.617808 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:16.617761    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:16.617808 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:16.617779    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ddd5cfc57-4mhc2: secret "image-registry-tls" not found
Jul 27 17:29:16.617948 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:16.617832    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls podName:b5ce3636-8fba-4864-99f7-28f1edd98f4e nodeName:}" failed. No retries permitted until 2026-07-27 17:29:20.617813785 +0000 UTC m=+42.662551864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls") pod "image-registry-5ddd5cfc57-4mhc2" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e") : secret "image-registry-tls" not found
Jul 27 17:29:16.639050 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:16.638087    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xdrd5" podStartSLOduration=8.118325405 podStartE2EDuration="38.638066985s" totalImagesPullingTime="30.51974158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:16.636185067 +0000 UTC m=+38.680923149" watchObservedRunningTime="2026-07-27 17:29:16.638066985 +0000 UTC m=+38.682805069"
Jul 27 17:29:18.619782 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:18.619745    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dvrgq" event={"ID":"f6825326-7c3b-4465-bc99-8f541ccad67f","Type":"ContainerStarted","Data":"1e9306e01f3cecf84cbfdeae726cd889460ea410687baff2978dc160490273da"}
Jul 27 17:29:18.638330 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:18.638212    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dvrgq" podStartSLOduration=1.786740001 podStartE2EDuration="6.638191591s" totalImagesPullingTime="4.85145159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:18.637838972 +0000 UTC m=+40.682577054" watchObservedRunningTime="2026-07-27 17:29:18.638191591 +0000 UTC m=+40.682929673"
Jul 27 17:29:19.646897 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.646865    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p"]
Jul 27 17:29:19.668204 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.668178    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"]
Jul 27 17:29:19.668379 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.668319    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p"
Jul 27 17:29:19.670932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.670912    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-25nh7\""
Jul 27 17:29:19.671212 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.671194    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 17:29:19.671295 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.671200    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 17:29:19.671344 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.671225    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 17:29:19.672140 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.672124    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 27 17:29:19.686193 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.686172    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p"]
Jul 27 17:29:19.686193 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.686197    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"]
Jul 27 17:29:19.686341 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.686273    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.689882 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.689563    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 27 17:29:19.689882 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.689621    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 27 17:29:19.689882 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.689643    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 27 17:29:19.689882 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.689569    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 27 17:29:19.844789 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.844751    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3e362be5-ec40-471a-8465-48cdf8f30eda-ca\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.844789 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.844792    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3e362be5-ec40-471a-8465-48cdf8f30eda-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.845021 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.844821    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e362be5-ec40-471a-8465-48cdf8f30eda-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.845021 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.844860    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e362be5-ec40-471a-8465-48cdf8f30eda-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.845021 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.844883    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwprd\" (UniqueName: \"kubernetes.io/projected/3e362be5-ec40-471a-8465-48cdf8f30eda-kube-api-access-kwprd\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.845021 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.844911    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3e362be5-ec40-471a-8465-48cdf8f30eda-hub\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.845021 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.844935    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wcsf\" (UniqueName: \"kubernetes.io/projected/d7399449-a96d-4790-94f8-88b4cb87533f-kube-api-access-9wcsf\") pod \"managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p\" (UID: \"d7399449-a96d-4790-94f8-88b4cb87533f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p"
Jul 27 17:29:19.845021 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.844974    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7399449-a96d-4790-94f8-88b4cb87533f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p\" (UID: \"d7399449-a96d-4790-94f8-88b4cb87533f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p"
Jul 27 17:29:19.946112 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.946035    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3e362be5-ec40-471a-8465-48cdf8f30eda-ca\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.946112 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.946072    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3e362be5-ec40-471a-8465-48cdf8f30eda-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.946112 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.946091    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e362be5-ec40-471a-8465-48cdf8f30eda-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.946354 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.946119    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e362be5-ec40-471a-8465-48cdf8f30eda-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.946354 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.946161    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwprd\" (UniqueName: \"kubernetes.io/projected/3e362be5-ec40-471a-8465-48cdf8f30eda-kube-api-access-kwprd\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.946354 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.946207    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3e362be5-ec40-471a-8465-48cdf8f30eda-hub\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.946354 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.946236    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wcsf\" (UniqueName: \"kubernetes.io/projected/d7399449-a96d-4790-94f8-88b4cb87533f-kube-api-access-9wcsf\") pod \"managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p\" (UID: \"d7399449-a96d-4790-94f8-88b4cb87533f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p"
Jul 27 17:29:19.946563 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.946445    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7399449-a96d-4790-94f8-88b4cb87533f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p\" (UID: \"d7399449-a96d-4790-94f8-88b4cb87533f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p"
Jul 27 17:29:19.946908 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.946886    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e362be5-ec40-471a-8465-48cdf8f30eda-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.950046 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.950019    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3e362be5-ec40-471a-8465-48cdf8f30eda-ca\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.950046 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.950035    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3e362be5-ec40-471a-8465-48cdf8f30eda-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.950200 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.950060    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3e362be5-ec40-471a-8465-48cdf8f30eda-hub\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.950200 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.950092    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e362be5-ec40-471a-8465-48cdf8f30eda-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.950200 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.950119    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7399449-a96d-4790-94f8-88b4cb87533f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p\" (UID: \"d7399449-a96d-4790-94f8-88b4cb87533f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p"
Jul 27 17:29:19.954100 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.954077    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwprd\" (UniqueName: \"kubernetes.io/projected/3e362be5-ec40-471a-8465-48cdf8f30eda-kube-api-access-kwprd\") pod \"cluster-proxy-proxy-agent-85db9948db-xhzxm\" (UID: \"3e362be5-ec40-471a-8465-48cdf8f30eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:19.954224 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.954208    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wcsf\" (UniqueName: \"kubernetes.io/projected/d7399449-a96d-4790-94f8-88b4cb87533f-kube-api-access-9wcsf\") pod \"managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p\" (UID: \"d7399449-a96d-4790-94f8-88b4cb87533f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p"
Jul 27 17:29:19.992196 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.992174    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p"
Jul 27 17:29:19.997279 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:19.997239    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:29:20.154744 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:20.154718    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p"]
Jul 27 17:29:20.157856 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:29:20.157830    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7399449_a96d_4790_94f8_88b4cb87533f.slice/crio-64514819f553cb5492f3276285ba64d9162db02a5ada1eb3ea121aef8b7808a1 WatchSource:0}: Error finding container 64514819f553cb5492f3276285ba64d9162db02a5ada1eb3ea121aef8b7808a1: Status 404 returned error can't find the container with id 64514819f553cb5492f3276285ba64d9162db02a5ada1eb3ea121aef8b7808a1
Jul 27 17:29:20.171097 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:20.171073    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"]
Jul 27 17:29:20.173958 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:29:20.173936    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e362be5_ec40_471a_8465_48cdf8f30eda.slice/crio-598409c9010391d1db02b3df795a22429f20b3d3ba9cb4caf275b8ae796262d5 WatchSource:0}: Error finding container 598409c9010391d1db02b3df795a22429f20b3d3ba9cb4caf275b8ae796262d5: Status 404 returned error can't find the container with id 598409c9010391d1db02b3df795a22429f20b3d3ba9cb4caf275b8ae796262d5
Jul 27 17:29:20.625634 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:20.625598    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p" event={"ID":"d7399449-a96d-4790-94f8-88b4cb87533f","Type":"ContainerStarted","Data":"64514819f553cb5492f3276285ba64d9162db02a5ada1eb3ea121aef8b7808a1"}
Jul 27 17:29:20.626618 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:20.626582    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm" event={"ID":"3e362be5-ec40-471a-8465-48cdf8f30eda","Type":"ContainerStarted","Data":"598409c9010391d1db02b3df795a22429f20b3d3ba9cb4caf275b8ae796262d5"}
Jul 27 17:29:20.653028 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:20.653000    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:20.653323 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:20.653045    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:20.653323 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:20.653144    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:20.653323 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:20.653165    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ddd5cfc57-4mhc2: secret "image-registry-tls" not found
Jul 27 17:29:20.653323 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:20.653148    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:20.653323 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:20.653215    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls podName:b5ce3636-8fba-4864-99f7-28f1edd98f4e nodeName:}" failed. No retries permitted until 2026-07-27 17:29:28.653200817 +0000 UTC m=+50.697938876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls") pod "image-registry-5ddd5cfc57-4mhc2" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e") : secret "image-registry-tls" not found
Jul 27 17:29:20.653323 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:20.653228    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls podName:b4177ef3-39b7-4328-9299-d1acac07b82d nodeName:}" failed. No retries permitted until 2026-07-27 17:29:28.653222319 +0000 UTC m=+50.697960377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls") pod "dns-default-zfdj9" (UID: "b4177ef3-39b7-4328-9299-d1acac07b82d") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:25.642552 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:25.642515    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm" event={"ID":"3e362be5-ec40-471a-8465-48cdf8f30eda","Type":"ContainerStarted","Data":"aef40b40b9294ca96e862e8065d977b94ba057e83b49fe128bcc8cc14b69f213"}
Jul 27 17:29:25.642552 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:25.642555    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm" event={"ID":"3e362be5-ec40-471a-8465-48cdf8f30eda","Type":"ContainerStarted","Data":"246be53c8c8dcd7feffd649a2ad9df0ccefbdc2cd7749139d98c459b6a424233"}
Jul 27 17:29:25.643083 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:25.642569    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm" event={"ID":"3e362be5-ec40-471a-8465-48cdf8f30eda","Type":"ContainerStarted","Data":"67fa118ba7f2052a66d138d54c6d68e4774cd3b03ca76468a6499d312eb4c5dc"}
Jul 27 17:29:25.643805 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:25.643783    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p" event={"ID":"d7399449-a96d-4790-94f8-88b4cb87533f","Type":"ContainerStarted","Data":"de4cfef40725eb033dec3f88021e090bb4e0c8c33138a40bec090c648e0404aa"}
Jul 27 17:29:25.662931 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:25.662888    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm" podStartSLOduration=2.177297898 podStartE2EDuration="6.662875484s" totalImagesPullingTime="4.485577586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:25.662300825 +0000 UTC m=+47.707038959" watchObservedRunningTime="2026-07-27 17:29:25.662875484 +0000 UTC m=+47.707613564"
Jul 27 17:29:25.676636 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:25.676592    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p" podStartSLOduration=2.190395727 podStartE2EDuration="6.676573759s" totalImagesPullingTime="4.486178032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:25.676476138 +0000 UTC m=+47.721214219" watchObservedRunningTime="2026-07-27 17:29:25.676573759 +0000 UTC m=+47.721311843"
Jul 27 17:29:28.712311 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:28.712269    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:28.712718 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:28.712320    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:28.712718 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:28.712424    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:28.712718 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:28.712484    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls podName:b4177ef3-39b7-4328-9299-d1acac07b82d nodeName:}" failed. No retries permitted until 2026-07-27 17:29:44.712471015 +0000 UTC m=+66.757209088 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls") pod "dns-default-zfdj9" (UID: "b4177ef3-39b7-4328-9299-d1acac07b82d") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:28.712718 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:28.712423    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:28.712718 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:28.712519    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ddd5cfc57-4mhc2: secret "image-registry-tls" not found
Jul 27 17:29:28.712718 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:28.712579    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls podName:b5ce3636-8fba-4864-99f7-28f1edd98f4e nodeName:}" failed. No retries permitted until 2026-07-27 17:29:44.712564086 +0000 UTC m=+66.757302164 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls") pod "image-registry-5ddd5cfc57-4mhc2" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e") : secret "image-registry-tls" not found
Jul 27 17:29:39.600624 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:39.600594    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-74rhp"
Jul 27 17:29:44.125892 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.125845    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:29:44.125892 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.125897    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:29:44.128686 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.128666    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 17:29:44.128754 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.128667    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 17:29:44.136921 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:44.136899    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 17:29:44.137014 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:44.136948    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs podName:50fb06f3-ab71-43cc-aa9c-58ba1b7651d6 nodeName:}" failed. No retries permitted until 2026-07-27 17:30:48.136935079 +0000 UTC m=+130.181673139 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs") pod "network-metrics-daemon-vtptl" (UID: "50fb06f3-ab71-43cc-aa9c-58ba1b7651d6") : secret "metrics-daemon-secret" not found
Jul 27 17:29:44.138910 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.138886    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbe546c0-bfc4-4692-a40b-3f0d22c0760d-original-pull-secret\") pod \"global-pull-secret-syncer-8kxfj\" (UID: \"cbe546c0-bfc4-4692-a40b-3f0d22c0760d\") " pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:29:44.226625 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.226589    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmlz\" (UniqueName: \"kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz\") pod \"network-check-target-7qfvm\" (UID: \"6bb08eb3-0b94-49f8-8c53-650e6ae67145\") " pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:29:44.229778 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.229750    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 17:29:44.239471 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.239452    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 17:29:44.249829 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.249812    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcmlz\" (UniqueName: \"kubernetes.io/projected/6bb08eb3-0b94-49f8-8c53-650e6ae67145-kube-api-access-rcmlz\") pod \"network-check-target-7qfvm\" (UID: \"6bb08eb3-0b94-49f8-8c53-650e6ae67145\") " pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:29:44.311025 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.310995    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8kxfj"
Jul 27 17:29:44.320006 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.319978    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zkh28\""
Jul 27 17:29:44.327608 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.327589    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:29:44.438421 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.438390    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8kxfj"]
Jul 27 17:29:44.440861 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:29:44.440838    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe546c0_bfc4_4692_a40b_3f0d22c0760d.slice/crio-b95cea2be4d4ae6bf1bde4ac2d5e691e31e86f7e5211bb3ad3984458a7f930fb WatchSource:0}: Error finding container b95cea2be4d4ae6bf1bde4ac2d5e691e31e86f7e5211bb3ad3984458a7f930fb: Status 404 returned error can't find the container with id b95cea2be4d4ae6bf1bde4ac2d5e691e31e86f7e5211bb3ad3984458a7f930fb
Jul 27 17:29:44.453339 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.453314    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7qfvm"]
Jul 27 17:29:44.456031 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:29:44.456008    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb08eb3_0b94_49f8_8c53_650e6ae67145.slice/crio-0d31458f9d1b031d6b25e5bc0736c99e6ba7c44f2837c6a3ebfcd38d3f71969b WatchSource:0}: Error finding container 0d31458f9d1b031d6b25e5bc0736c99e6ba7c44f2837c6a3ebfcd38d3f71969b: Status 404 returned error can't find the container with id 0d31458f9d1b031d6b25e5bc0736c99e6ba7c44f2837c6a3ebfcd38d3f71969b
Jul 27 17:29:44.688427 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.688352    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7qfvm" event={"ID":"6bb08eb3-0b94-49f8-8c53-650e6ae67145","Type":"ContainerStarted","Data":"0d31458f9d1b031d6b25e5bc0736c99e6ba7c44f2837c6a3ebfcd38d3f71969b"}
Jul 27 17:29:44.689408 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.689383    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8kxfj" event={"ID":"cbe546c0-bfc4-4692-a40b-3f0d22c0760d","Type":"ContainerStarted","Data":"b95cea2be4d4ae6bf1bde4ac2d5e691e31e86f7e5211bb3ad3984458a7f930fb"}
Jul 27 17:29:44.730460 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.730433    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:29:44.730558 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:44.730474    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:29:44.730616 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:44.730598    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:44.730650 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:44.730621    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ddd5cfc57-4mhc2: secret "image-registry-tls" not found
Jul 27 17:29:44.730686 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:44.730666    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls podName:b5ce3636-8fba-4864-99f7-28f1edd98f4e nodeName:}" failed. No retries permitted until 2026-07-27 17:30:16.730651598 +0000 UTC m=+98.775389670 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls") pod "image-registry-5ddd5cfc57-4mhc2" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e") : secret "image-registry-tls" not found
Jul 27 17:29:44.730726 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:44.730603    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:44.730760 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:29:44.730728    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls podName:b4177ef3-39b7-4328-9299-d1acac07b82d nodeName:}" failed. No retries permitted until 2026-07-27 17:30:16.730716058 +0000 UTC m=+98.775454119 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls") pod "dns-default-zfdj9" (UID: "b4177ef3-39b7-4328-9299-d1acac07b82d") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:49.705672 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:49.705632    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7qfvm" event={"ID":"6bb08eb3-0b94-49f8-8c53-650e6ae67145","Type":"ContainerStarted","Data":"d2b79290dfee69ff35543b232852b589f0ea342c5488ae30aa68aad944298068"}
Jul 27 17:29:49.706192 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:49.705710    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:29:49.707025 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:49.706997    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8kxfj" event={"ID":"cbe546c0-bfc4-4692-a40b-3f0d22c0760d","Type":"ContainerStarted","Data":"9defc39eee93ca446dfe49f3090c906757f07cc6dcc647c136fe6c2fa7b8862e"}
Jul 27 17:29:49.720642 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:49.720595    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7qfvm" podStartSLOduration=67.108742625 podStartE2EDuration="1m11.720584054s" totalImagesPullingTime="4.611841429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:49.720367788 +0000 UTC m=+71.765105870" watchObservedRunningTime="2026-07-27 17:29:49.720584054 +0000 UTC m=+71.765322132"
Jul 27 17:29:49.734407 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:29:49.734368    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8kxfj" podStartSLOduration=67.103644196 podStartE2EDuration="1m11.734355697s" totalImagesPullingTime="4.630711501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:49.733728871 +0000 UTC m=+71.778466956" watchObservedRunningTime="2026-07-27 17:29:49.734355697 +0000 UTC m=+71.779093778"
Jul 27 17:30:16.768590 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:30:16.768550    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:30:16.768590 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:30:16.768596    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:30:16.769129 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:30:16.768684    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:30:16.769129 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:30:16.768690    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:30:16.769129 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:30:16.768709    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ddd5cfc57-4mhc2: secret "image-registry-tls" not found
Jul 27 17:30:16.769129 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:30:16.768743    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls podName:b4177ef3-39b7-4328-9299-d1acac07b82d nodeName:}" failed. No retries permitted until 2026-07-27 17:31:20.768728453 +0000 UTC m=+162.813466525 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls") pod "dns-default-zfdj9" (UID: "b4177ef3-39b7-4328-9299-d1acac07b82d") : secret "dns-default-metrics-tls" not found
Jul 27 17:30:16.769129 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:30:16.768756    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls podName:b5ce3636-8fba-4864-99f7-28f1edd98f4e nodeName:}" failed. No retries permitted until 2026-07-27 17:31:20.76875003 +0000 UTC m=+162.813488089 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls") pod "image-registry-5ddd5cfc57-4mhc2" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e") : secret "image-registry-tls" not found
Jul 27 17:30:20.712598 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:30:20.712568    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7qfvm"
Jul 27 17:30:48.188477 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:30:48.188436    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:30:48.188998 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:30:48.188582    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 17:30:48.188998 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:30:48.188656    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs podName:50fb06f3-ab71-43cc-aa9c-58ba1b7651d6 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:50.18864107 +0000 UTC m=+252.233379134 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs") pod "network-metrics-daemon-vtptl" (UID: "50fb06f3-ab71-43cc-aa9c-58ba1b7651d6") : secret "metrics-daemon-secret" not found
Jul 27 17:31:15.858923 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:31:15.858872    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2" podUID="b5ce3636-8fba-4864-99f7-28f1edd98f4e"
Jul 27 17:31:15.872030 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:31:15.871995    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zfdj9" podUID="b4177ef3-39b7-4328-9299-d1acac07b82d"
Jul 27 17:31:15.947340 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:31:15.947312    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:31:15.947514 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:31:15.947312    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:31:16.404608 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:31:16.404572    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:31:20.812832 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:31:20.812793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls\") pod \"image-registry-5ddd5cfc57-4mhc2\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") " pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:31:20.813214 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:31:20.812840    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:31:20.813214 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:31:20.812950    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:31:20.813214 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:31:20.812969    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ddd5cfc57-4mhc2: secret "image-registry-tls" not found
Jul 27 17:31:20.813214 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:31:20.813011    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:31:20.813214 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:31:20.813036    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls podName:b5ce3636-8fba-4864-99f7-28f1edd98f4e nodeName:}" failed. No retries permitted until 2026-07-27 17:33:22.813019023 +0000 UTC m=+284.857757111 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls") pod "image-registry-5ddd5cfc57-4mhc2" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e") : secret "image-registry-tls" not found
Jul 27 17:31:20.813214 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:31:20.813058    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls podName:b4177ef3-39b7-4328-9299-d1acac07b82d nodeName:}" failed. No retries permitted until 2026-07-27 17:33:22.813046977 +0000 UTC m=+284.857785055 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls") pod "dns-default-zfdj9" (UID: "b4177ef3-39b7-4328-9299-d1acac07b82d") : secret "dns-default-metrics-tls" not found
Jul 27 17:31:24.974892 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:31:24.974859    2571 generic.go:356] "Generic (PLEG): container finished" podID="d7399449-a96d-4790-94f8-88b4cb87533f" containerID="de4cfef40725eb033dec3f88021e090bb4e0c8c33138a40bec090c648e0404aa" exitCode=255
Jul 27 17:31:24.974892 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:31:24.974903    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p" event={"ID":"d7399449-a96d-4790-94f8-88b4cb87533f","Type":"ContainerDied","Data":"de4cfef40725eb033dec3f88021e090bb4e0c8c33138a40bec090c648e0404aa"}
Jul 27 17:31:24.975294 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:31:24.975152    2571 scope.go:117] "RemoveContainer" containerID="de4cfef40725eb033dec3f88021e090bb4e0c8c33138a40bec090c648e0404aa"
Jul 27 17:31:25.979253 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:31:25.979222    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p" event={"ID":"d7399449-a96d-4790-94f8-88b4cb87533f","Type":"ContainerStarted","Data":"7af5e33a6c8950d1ecbbc8eaa1d3bf635b1778f725cf501c22cec642d0c02720"}
Jul 27 17:31:28.394912 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:31:28.394879    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:32:42.737303 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.737274    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dd9n4"]
Jul 27 17:32:42.740181 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.740160    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:42.742698 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.742676    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 17:32:42.743827 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.743810    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 17:32:42.743827 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.743824    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 17:32:42.743982 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.743831    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 17:32:42.743982 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.743835    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8r9mw\""
Jul 27 17:32:42.752328 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.752308    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dd9n4"]
Jul 27 17:32:42.798107 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.798070    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2lm5\" (UniqueName: \"kubernetes.io/projected/f514ad0b-436a-4636-a106-63be09beab94-kube-api-access-z2lm5\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:42.798107 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.798105    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f514ad0b-436a-4636-a106-63be09beab94-data-volume\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:42.798296 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.798145    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:42.798296 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.798165    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f514ad0b-436a-4636-a106-63be09beab94-crio-socket\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:42.798296 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.798191    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:42.899447 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.899411    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:42.899447 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.899453    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f514ad0b-436a-4636-a106-63be09beab94-crio-socket\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:42.899688 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.899485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:42.899688 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.899539    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2lm5\" (UniqueName: \"kubernetes.io/projected/f514ad0b-436a-4636-a106-63be09beab94-kube-api-access-z2lm5\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:42.899688 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:42.899544    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:32:42.899688 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.899564    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f514ad0b-436a-4636-a106-63be09beab94-data-volume\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:42.899688 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.899605    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f514ad0b-436a-4636-a106-63be09beab94-crio-socket\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:42.899688 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:42.899627    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm podName:f514ad0b-436a-4636-a106-63be09beab94 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:43.399605556 +0000 UTC m=+245.444343617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dd9n4" (UID: "f514ad0b-436a-4636-a106-63be09beab94") : configmap "kube-rbac-proxy" not found
Jul 27 17:32:42.899688 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:42.899677    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:42.899988 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:42.899754    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls podName:f514ad0b-436a-4636-a106-63be09beab94 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:43.39973572 +0000 UTC m=+245.444473781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd9n4" (UID: "f514ad0b-436a-4636-a106-63be09beab94") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:42.899988 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.899875    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f514ad0b-436a-4636-a106-63be09beab94-data-volume\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:42.907952 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:42.907934    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2lm5\" (UniqueName: \"kubernetes.io/projected/f514ad0b-436a-4636-a106-63be09beab94-kube-api-access-z2lm5\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:43.403754 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:43.403716    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:43.403925 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:43.403768    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:43.403925 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:43.403830    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:32:43.403925 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:43.403846    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:43.403925 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:43.403899    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm podName:f514ad0b-436a-4636-a106-63be09beab94 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:44.403884089 +0000 UTC m=+246.448622148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dd9n4" (UID: "f514ad0b-436a-4636-a106-63be09beab94") : configmap "kube-rbac-proxy" not found
Jul 27 17:32:43.403925 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:43.403913    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls podName:f514ad0b-436a-4636-a106-63be09beab94 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:44.403906736 +0000 UTC m=+246.448644795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd9n4" (UID: "f514ad0b-436a-4636-a106-63be09beab94") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:44.413109 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:44.413078    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:44.413478 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:44.413160    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:44.413478 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:44.413216    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:44.413478 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:44.413239    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:32:44.413478 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:44.413273    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls podName:f514ad0b-436a-4636-a106-63be09beab94 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:46.413257495 +0000 UTC m=+248.457995554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd9n4" (UID: "f514ad0b-436a-4636-a106-63be09beab94") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:44.413478 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:44.413287    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm podName:f514ad0b-436a-4636-a106-63be09beab94 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:46.41328085 +0000 UTC m=+248.458018909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dd9n4" (UID: "f514ad0b-436a-4636-a106-63be09beab94") : configmap "kube-rbac-proxy" not found
Jul 27 17:32:44.505745 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:44.505716    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8lrn8_f50b4a82-db6d-409a-8c83-eb229b3e2e13/dns-node-resolver/0.log"
Jul 27 17:32:45.706519 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:45.706476    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ckvrn_367e4815-f156-4241-88f1-b8e84258fd04/node-ca/0.log"
Jul 27 17:32:46.427817 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:46.427786    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:46.427966 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:46.427837    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:46.427966 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:46.427897    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:32:46.427966 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:46.427955    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm podName:f514ad0b-436a-4636-a106-63be09beab94 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:50.427941137 +0000 UTC m=+252.472679196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dd9n4" (UID: "f514ad0b-436a-4636-a106-63be09beab94") : configmap "kube-rbac-proxy" not found
Jul 27 17:32:46.427966 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:46.427960    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:46.428126 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:46.428001    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls podName:f514ad0b-436a-4636-a106-63be09beab94 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:50.427989378 +0000 UTC m=+252.472727437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd9n4" (UID: "f514ad0b-436a-4636-a106-63be09beab94") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:46.505911 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:46.505880    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dvrgq_f6825326-7c3b-4465-bc99-8f541ccad67f/serve-healthcheck-canary/0.log"
Jul 27 17:32:50.253529 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:50.253482    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:32:50.253913 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:50.253634    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 17:32:50.253913 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:50.253702    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs podName:50fb06f3-ab71-43cc-aa9c-58ba1b7651d6 nodeName:}" failed. No retries permitted until 2026-07-27 17:34:52.253687906 +0000 UTC m=+374.298425965 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs") pod "network-metrics-daemon-vtptl" (UID: "50fb06f3-ab71-43cc-aa9c-58ba1b7651d6") : secret "metrics-daemon-secret" not found
Jul 27 17:32:50.454254 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:50.454222    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:50.454420 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:50.454273    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:50.454420 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:50.454340    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:32:50.454420 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:50.454360    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:50.454420 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:50.454400    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm podName:f514ad0b-436a-4636-a106-63be09beab94 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:58.454384399 +0000 UTC m=+260.499122457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dd9n4" (UID: "f514ad0b-436a-4636-a106-63be09beab94") : configmap "kube-rbac-proxy" not found
Jul 27 17:32:50.454420 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:32:50.454414    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls podName:f514ad0b-436a-4636-a106-63be09beab94 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:58.454408079 +0000 UTC m=+260.499146139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dd9n4" (UID: "f514ad0b-436a-4636-a106-63be09beab94") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:58.513390 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:58.513353    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:58.513768 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:58.513414    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:58.513844 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:58.513823    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f514ad0b-436a-4636-a106-63be09beab94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:58.515629 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:58.515614    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f514ad0b-436a-4636-a106-63be09beab94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dd9n4\" (UID: \"f514ad0b-436a-4636-a106-63be09beab94\") " pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:58.649444 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:58.649408    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dd9n4"
Jul 27 17:32:58.768047 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:58.767976    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dd9n4"]
Jul 27 17:32:58.771009 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:32:58.770985    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf514ad0b_436a_4636_a106_63be09beab94.slice/crio-ac090b5837aa537a06b8132c203af9dbef2cd9d258f91027d987c2c67194dcbd WatchSource:0}: Error finding container ac090b5837aa537a06b8132c203af9dbef2cd9d258f91027d987c2c67194dcbd: Status 404 returned error can't find the container with id ac090b5837aa537a06b8132c203af9dbef2cd9d258f91027d987c2c67194dcbd
Jul 27 17:32:59.232629 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:32:59.232595    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dd9n4" event={"ID":"f514ad0b-436a-4636-a106-63be09beab94","Type":"ContainerStarted","Data":"ac090b5837aa537a06b8132c203af9dbef2cd9d258f91027d987c2c67194dcbd"}
Jul 27 17:33:04.248990 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:04.248958    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dd9n4" event={"ID":"f514ad0b-436a-4636-a106-63be09beab94","Type":"ContainerStarted","Data":"fcd332e3b1203b19f7b8ce2a872626c8913045239d1ee656d7269faf7d9863a4"}
Jul 27 17:33:06.626512 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:06.626468    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"]
Jul 27 17:33:06.626954 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:33:06.626717    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2" podUID="b5ce3636-8fba-4864-99f7-28f1edd98f4e"
Jul 27 17:33:07.257917 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.257880    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:33:07.280171 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.280142    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:33:07.381742 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.381373    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5ce3636-8fba-4864-99f7-28f1edd98f4e-installation-pull-secrets\") pod \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") "
Jul 27 17:33:07.381742 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.381439    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-certificates\") pod \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") "
Jul 27 17:33:07.381742 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.381466    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5ce3636-8fba-4864-99f7-28f1edd98f4e-ca-trust-extracted\") pod \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") "
Jul 27 17:33:07.381742 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.381519    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfp29\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-kube-api-access-vfp29\") pod \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") "
Jul 27 17:33:07.381742 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.381561    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5ce3636-8fba-4864-99f7-28f1edd98f4e-trusted-ca\") pod \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") "
Jul 27 17:33:07.381742 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.381605    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-bound-sa-token\") pod \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") "
Jul 27 17:33:07.381742 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.381640    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5ce3636-8fba-4864-99f7-28f1edd98f4e-image-registry-private-configuration\") pod \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\" (UID: \"b5ce3636-8fba-4864-99f7-28f1edd98f4e\") "
Jul 27 17:33:07.382236 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.382154    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b5ce3636-8fba-4864-99f7-28f1edd98f4e" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:33:07.382236 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.382166    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ce3636-8fba-4864-99f7-28f1edd98f4e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b5ce3636-8fba-4864-99f7-28f1edd98f4e" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:33:07.382344 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.382247    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ce3636-8fba-4864-99f7-28f1edd98f4e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b5ce3636-8fba-4864-99f7-28f1edd98f4e" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:33:07.384195 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.384163    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5ce3636-8fba-4864-99f7-28f1edd98f4e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b5ce3636-8fba-4864-99f7-28f1edd98f4e" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:33:07.384338 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.384217    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-kube-api-access-vfp29" (OuterVolumeSpecName: "kube-api-access-vfp29") pod "b5ce3636-8fba-4864-99f7-28f1edd98f4e" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e"). InnerVolumeSpecName "kube-api-access-vfp29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:33:07.384338 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.384329    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5ce3636-8fba-4864-99f7-28f1edd98f4e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b5ce3636-8fba-4864-99f7-28f1edd98f4e" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:33:07.384725 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.384687    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b5ce3636-8fba-4864-99f7-28f1edd98f4e" (UID: "b5ce3636-8fba-4864-99f7-28f1edd98f4e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:33:07.483285 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.483238    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-certificates\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:33:07.483285 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.483269    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5ce3636-8fba-4864-99f7-28f1edd98f4e-ca-trust-extracted\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:33:07.483285 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.483282    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfp29\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-kube-api-access-vfp29\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:33:07.483285 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.483295    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5ce3636-8fba-4864-99f7-28f1edd98f4e-trusted-ca\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:33:07.483637 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.483309    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-bound-sa-token\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:33:07.483637 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.483321    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5ce3636-8fba-4864-99f7-28f1edd98f4e-image-registry-private-configuration\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:33:07.483637 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:07.483335    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5ce3636-8fba-4864-99f7-28f1edd98f4e-installation-pull-secrets\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:33:08.260891 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:08.260861    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"
Jul 27 17:33:08.294719 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:08.294693    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"]
Jul 27 17:33:08.298226 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:08.298202    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5ddd5cfc57-4mhc2"]
Jul 27 17:33:08.391137 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:08.391115    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5ce3636-8fba-4864-99f7-28f1edd98f4e-registry-tls\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:33:08.396660 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:08.396640    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5ce3636-8fba-4864-99f7-28f1edd98f4e" path="/var/lib/kubelet/pods/b5ce3636-8fba-4864-99f7-28f1edd98f4e/volumes"
Jul 27 17:33:09.266099 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.266064    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dd9n4" event={"ID":"f514ad0b-436a-4636-a106-63be09beab94","Type":"ContainerStarted","Data":"45ef15c1c5a7edfe7b83e240f84ed38d2f99591d1b8b3457a68ab6d3569636c4"}
Jul 27 17:33:09.566849 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.566766    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-79npr"]
Jul 27 17:33:09.569851 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.569827    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:09.572621 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.572594    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 17:33:09.574179 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.573972    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 17:33:09.574179 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.573980    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 17:33:09.574179 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.574005    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mpmfm\""
Jul 27 17:33:09.574179 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.574035    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 17:33:09.574471 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.574388    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 17:33:09.581779 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.581759    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-79npr"]
Jul 27 17:33:09.701152 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.701109    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba12ae86-da35-4e87-8fce-99b209e13ee8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-79npr\" (UID: \"ba12ae86-da35-4e87-8fce-99b209e13ee8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:09.701334 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.701196    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba12ae86-da35-4e87-8fce-99b209e13ee8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-79npr\" (UID: \"ba12ae86-da35-4e87-8fce-99b209e13ee8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:09.701334 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.701227    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xpkc\" (UniqueName: \"kubernetes.io/projected/ba12ae86-da35-4e87-8fce-99b209e13ee8-kube-api-access-4xpkc\") pod \"prometheus-operator-84d69f794d-79npr\" (UID: \"ba12ae86-da35-4e87-8fce-99b209e13ee8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:09.701334 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.701282    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba12ae86-da35-4e87-8fce-99b209e13ee8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-79npr\" (UID: \"ba12ae86-da35-4e87-8fce-99b209e13ee8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:09.801785 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.801748    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba12ae86-da35-4e87-8fce-99b209e13ee8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-79npr\" (UID: \"ba12ae86-da35-4e87-8fce-99b209e13ee8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:09.801785 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.801785    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba12ae86-da35-4e87-8fce-99b209e13ee8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-79npr\" (UID: \"ba12ae86-da35-4e87-8fce-99b209e13ee8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:09.802032 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.801868    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba12ae86-da35-4e87-8fce-99b209e13ee8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-79npr\" (UID: \"ba12ae86-da35-4e87-8fce-99b209e13ee8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:09.802032 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.801897    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpkc\" (UniqueName: \"kubernetes.io/projected/ba12ae86-da35-4e87-8fce-99b209e13ee8-kube-api-access-4xpkc\") pod \"prometheus-operator-84d69f794d-79npr\" (UID: \"ba12ae86-da35-4e87-8fce-99b209e13ee8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:09.802911 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.802878    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba12ae86-da35-4e87-8fce-99b209e13ee8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-79npr\" (UID: \"ba12ae86-da35-4e87-8fce-99b209e13ee8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:09.804534 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.804488    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba12ae86-da35-4e87-8fce-99b209e13ee8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-79npr\" (UID: \"ba12ae86-da35-4e87-8fce-99b209e13ee8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:09.804679 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.804647    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba12ae86-da35-4e87-8fce-99b209e13ee8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-79npr\" (UID: \"ba12ae86-da35-4e87-8fce-99b209e13ee8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:09.810609 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.810585    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xpkc\" (UniqueName: \"kubernetes.io/projected/ba12ae86-da35-4e87-8fce-99b209e13ee8-kube-api-access-4xpkc\") pod \"prometheus-operator-84d69f794d-79npr\" (UID: \"ba12ae86-da35-4e87-8fce-99b209e13ee8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:09.880450 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:09.880369    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr"
Jul 27 17:33:10.020877 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:10.020799    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-79npr"]
Jul 27 17:33:10.024791 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:33:10.024762    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba12ae86_da35_4e87_8fce_99b209e13ee8.slice/crio-29f080b9c107e2f4276139845d1e0a3ad078cb1bc75e370732438e0ce8851f40 WatchSource:0}: Error finding container 29f080b9c107e2f4276139845d1e0a3ad078cb1bc75e370732438e0ce8851f40: Status 404 returned error can't find the container with id 29f080b9c107e2f4276139845d1e0a3ad078cb1bc75e370732438e0ce8851f40
Jul 27 17:33:10.271153 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:10.271120    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr" event={"ID":"ba12ae86-da35-4e87-8fce-99b209e13ee8","Type":"ContainerStarted","Data":"29f080b9c107e2f4276139845d1e0a3ad078cb1bc75e370732438e0ce8851f40"}
Jul 27 17:33:12.279913 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:12.279875    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dd9n4" event={"ID":"f514ad0b-436a-4636-a106-63be09beab94","Type":"ContainerStarted","Data":"cd3e78c22a73f6df41d1d6966d1968bc75467a29dd05b2ca8c7ce82b6174fb2e"}
Jul 27 17:33:12.298086 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:12.298035    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dd9n4" podStartSLOduration=17.445920195 podStartE2EDuration="30.298017229s" totalImagesPullingTime="12.852097034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:32:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:12.297183354 +0000 UTC m=+274.341921437" watchObservedRunningTime="2026-07-27 17:33:12.298017229 +0000 UTC m=+274.342755309"
Jul 27 17:33:13.284909 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:13.284866    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr" event={"ID":"ba12ae86-da35-4e87-8fce-99b209e13ee8","Type":"ContainerStarted","Data":"a044316ad947f39454fd7b5e2e29928c4d4fb2d9f54e5ca9c6dfa89889ea4cf6"}
Jul 27 17:33:13.285367 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:13.284965    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr" event={"ID":"ba12ae86-da35-4e87-8fce-99b209e13ee8","Type":"ContainerStarted","Data":"6abe20980bee6d0381d03cacf6ee0036727bfa25c2c324f17d6e767bfa4f574a"}
Jul 27 17:33:13.301331 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:13.301279    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-79npr" podStartSLOduration=1.656306189 podStartE2EDuration="4.301265513s" totalImagesPullingTime="2.644959324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:13.299735397 +0000 UTC m=+275.344473481" watchObservedRunningTime="2026-07-27 17:33:13.301265513 +0000 UTC m=+275.346003593"
Jul 27 17:33:18.947825 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:33:18.947782    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zfdj9" podUID="b4177ef3-39b7-4328-9299-d1acac07b82d"
Jul 27 17:33:19.302361 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.302326    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:33:19.822137 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.822073    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p64pw"]
Jul 27 17:33:19.825358 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.825337    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:19.827874 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.827855    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 17:33:19.827985 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.827887    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 17:33:19.827985 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.827934    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jrhwj\""
Jul 27 17:33:19.828429 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.828412    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 17:33:19.977899 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.977861    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-textfile\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:19.978365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.977916    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-metrics-client-ca\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:19.978365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.978038    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-wtmp\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:19.978365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.978084    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:19.978365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.978126    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-sys\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:19.978365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.978151    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4cd\" (UniqueName: \"kubernetes.io/projected/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-kube-api-access-2q4cd\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:19.978365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.978189    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-root\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:19.978365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.978212    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-tls\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:19.978365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.978234    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:19.998523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:19.998469    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm" podUID="3e362be5-ec40-471a-8465-48cdf8f30eda" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 27 17:33:20.079122 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.079044    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-wtmp\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.079122 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.079080    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.079122 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.079101    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-sys\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.079122 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.079117    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4cd\" (UniqueName: \"kubernetes.io/projected/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-kube-api-access-2q4cd\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.079405 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.079145    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-root\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.079405 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.079165    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-tls\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.079405 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.079190    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-sys\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.079405 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.079191    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.079405 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.079223    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-root\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.079405 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.079223    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-wtmp\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.079405 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.079228    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-textfile\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.079405 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.079317    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-metrics-client-ca\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.079405 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:33:20.079325    2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 17:33:20.079405 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:33:20.079397    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-tls podName:b1aa6edb-029a-49bb-8a2a-41e00c595b7b nodeName:}" failed. No retries permitted until 2026-07-27 17:33:20.579377155 +0000 UTC m=+282.624115217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-tls") pod "node-exporter-p64pw" (UID: "b1aa6edb-029a-49bb-8a2a-41e00c595b7b") : secret "node-exporter-tls" not found
Jul 27 17:33:20.080043 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.079864    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-textfile\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.080087 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.080070    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-metrics-client-ca\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.080126 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.080081    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.081697 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.081681    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.087092 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.087064    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4cd\" (UniqueName: \"kubernetes.io/projected/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-kube-api-access-2q4cd\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.585070 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.585035    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-tls\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.587276 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.587259    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1aa6edb-029a-49bb-8a2a-41e00c595b7b-node-exporter-tls\") pod \"node-exporter-p64pw\" (UID: \"b1aa6edb-029a-49bb-8a2a-41e00c595b7b\") " pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.734287 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:20.734251    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p64pw"
Jul 27 17:33:20.745189 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:33:20.745160    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1aa6edb_029a_49bb_8a2a_41e00c595b7b.slice/crio-ed8c815f3a1c70d4ea8753785082d0a255206e36a418cb402c365845ce62af22 WatchSource:0}: Error finding container ed8c815f3a1c70d4ea8753785082d0a255206e36a418cb402c365845ce62af22: Status 404 returned error can't find the container with id ed8c815f3a1c70d4ea8753785082d0a255206e36a418cb402c365845ce62af22
Jul 27 17:33:21.311273 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:21.311224    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p64pw" event={"ID":"b1aa6edb-029a-49bb-8a2a-41e00c595b7b","Type":"ContainerStarted","Data":"ed8c815f3a1c70d4ea8753785082d0a255206e36a418cb402c365845ce62af22"}
Jul 27 17:33:22.315600 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.315564    2571 generic.go:356] "Generic (PLEG): container finished" podID="b1aa6edb-029a-49bb-8a2a-41e00c595b7b" containerID="0f32667df8c9291fc8f3ed0acfd76bd63092b0ce15df17ab110b498eb94bb601" exitCode=0
Jul 27 17:33:22.316005 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.315653    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p64pw" event={"ID":"b1aa6edb-029a-49bb-8a2a-41e00c595b7b","Type":"ContainerDied","Data":"0f32667df8c9291fc8f3ed0acfd76bd63092b0ce15df17ab110b498eb94bb601"}
Jul 27 17:33:22.809569 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.809533    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f67999996-4qfvx"]
Jul 27 17:33:22.813008 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.812989    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:22.815750 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.815711    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 27 17:33:22.815844 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.815769    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1msa1ir846tj7\""
Jul 27 17:33:22.815844 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.815785    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m9g79\""
Jul 27 17:33:22.815932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.815785    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 27 17:33:22.815932 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.815770    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 27 17:33:22.816130 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.816114    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 27 17:33:22.816168 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.816159    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 27 17:33:22.823894 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.823837    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f67999996-4qfvx"]
Jul 27 17:33:22.899117 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.899086    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzjm2\" (UniqueName: \"kubernetes.io/projected/35036ce7-166a-4289-b4c2-73b08e3c282e-kube-api-access-vzjm2\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:22.899262 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.899125    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:22.899262 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.899153    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:33:22.899262 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.899203    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35036ce7-166a-4289-b4c2-73b08e3c282e-metrics-client-ca\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:22.899362 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.899276    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:22.899362 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.899302    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:22.899362 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.899334    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:22.899362 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.899360    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-grpc-tls\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:22.899482 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.899390    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-tls\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:22.901454 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.901429    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4177ef3-39b7-4328-9299-d1acac07b82d-metrics-tls\") pod \"dns-default-zfdj9\" (UID: \"b4177ef3-39b7-4328-9299-d1acac07b82d\") " pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:33:22.905733 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.905716    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x7wp7\""
Jul 27 17:33:22.913876 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:22.913860    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:33:23.000092 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.000048    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35036ce7-166a-4289-b4c2-73b08e3c282e-metrics-client-ca\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.000257 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.000097    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.000257 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.000136    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.000257 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.000163    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.000257 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.000185    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-grpc-tls\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.001476 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.000596    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-tls\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.001476 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.000698    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzjm2\" (UniqueName: \"kubernetes.io/projected/35036ce7-166a-4289-b4c2-73b08e3c282e-kube-api-access-vzjm2\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.001476 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.000752    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.001476 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.001386    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35036ce7-166a-4289-b4c2-73b08e3c282e-metrics-client-ca\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.012380 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.012009    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-tls\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.012998 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.012958    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.013814 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.013777    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.013814 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.013803    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-grpc-tls\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.014002 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.013831    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.014463 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.014438    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35036ce7-166a-4289-b4c2-73b08e3c282e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.015480 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.015445    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzjm2\" (UniqueName: \"kubernetes.io/projected/35036ce7-166a-4289-b4c2-73b08e3c282e-kube-api-access-vzjm2\") pod \"thanos-querier-7f67999996-4qfvx\" (UID: \"35036ce7-166a-4289-b4c2-73b08e3c282e\") " pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.043052 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.043025    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zfdj9"]
Jul 27 17:33:23.046776 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:33:23.046754    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4177ef3_39b7_4328_9299_d1acac07b82d.slice/crio-b298971621c138f3515d8264f6ee177c92b90a13ef27b0f58a1c575f03b08e91 WatchSource:0}: Error finding container b298971621c138f3515d8264f6ee177c92b90a13ef27b0f58a1c575f03b08e91: Status 404 returned error can't find the container with id b298971621c138f3515d8264f6ee177c92b90a13ef27b0f58a1c575f03b08e91
Jul 27 17:33:23.122281 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.122204    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:23.248201 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.248166    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f67999996-4qfvx"]
Jul 27 17:33:23.251684 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:33:23.251654    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35036ce7_166a_4289_b4c2_73b08e3c282e.slice/crio-67898cbfb495736276f141e0cbd98ab933eee7758bdc6ef3d4afeb3d180e7eab WatchSource:0}: Error finding container 67898cbfb495736276f141e0cbd98ab933eee7758bdc6ef3d4afeb3d180e7eab: Status 404 returned error can't find the container with id 67898cbfb495736276f141e0cbd98ab933eee7758bdc6ef3d4afeb3d180e7eab
Jul 27 17:33:23.319827 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.319796    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zfdj9" event={"ID":"b4177ef3-39b7-4328-9299-d1acac07b82d","Type":"ContainerStarted","Data":"b298971621c138f3515d8264f6ee177c92b90a13ef27b0f58a1c575f03b08e91"}
Jul 27 17:33:23.320950 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.320917    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx" event={"ID":"35036ce7-166a-4289-b4c2-73b08e3c282e","Type":"ContainerStarted","Data":"67898cbfb495736276f141e0cbd98ab933eee7758bdc6ef3d4afeb3d180e7eab"}
Jul 27 17:33:23.322899 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.322879    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p64pw" event={"ID":"b1aa6edb-029a-49bb-8a2a-41e00c595b7b","Type":"ContainerStarted","Data":"d5ded9578dd4e65a586a91c9635cfcc3ce1d69fa16f3e1eecd34b6a98e38e3af"}
Jul 27 17:33:23.323000 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.322916    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p64pw" event={"ID":"b1aa6edb-029a-49bb-8a2a-41e00c595b7b","Type":"ContainerStarted","Data":"c8998cc89f5e158fc58715db38e456daf151d1b4633ee2ae06dad8e920625996"}
Jul 27 17:33:23.341549 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:23.341488    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p64pw" podStartSLOduration=3.330384088 podStartE2EDuration="4.341472853s" totalImagesPullingTime="950.764952ms" totalInitContainerRuntime="60.323813ms" isStatefulPod=true podCreationTimestamp="2026-07-27 17:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:23.341377919 +0000 UTC m=+285.386116000" watchObservedRunningTime="2026-07-27 17:33:23.341472853 +0000 UTC m=+285.386210935"
Jul 27 17:33:25.332591 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:25.332514    2571 generic.go:356] "Generic (PLEG): container finished" podID="d7399449-a96d-4790-94f8-88b4cb87533f" containerID="7af5e33a6c8950d1ecbbc8eaa1d3bf635b1778f725cf501c22cec642d0c02720" exitCode=255
Jul 27 17:33:25.332977 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:25.332522    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p" event={"ID":"d7399449-a96d-4790-94f8-88b4cb87533f","Type":"ContainerDied","Data":"7af5e33a6c8950d1ecbbc8eaa1d3bf635b1778f725cf501c22cec642d0c02720"}
Jul 27 17:33:25.332977 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:25.332632    2571 scope.go:117] "RemoveContainer" containerID="de4cfef40725eb033dec3f88021e090bb4e0c8c33138a40bec090c648e0404aa"
Jul 27 17:33:25.332977 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:25.332950    2571 scope.go:117] "RemoveContainer" containerID="7af5e33a6c8950d1ecbbc8eaa1d3bf635b1778f725cf501c22cec642d0c02720"
Jul 27 17:33:25.333139 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:33:25.333121    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p_open-cluster-management-agent-addon(d7399449-a96d-4790-94f8-88b4cb87533f)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p" podUID="d7399449-a96d-4790-94f8-88b4cb87533f"
Jul 27 17:33:25.334337 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:25.334319    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zfdj9" event={"ID":"b4177ef3-39b7-4328-9299-d1acac07b82d","Type":"ContainerStarted","Data":"2a0c657e36309270e27bc19602e4f7bff838aa4809acf0717e398e0228c19dd4"}
Jul 27 17:33:25.334433 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:25.334363    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zfdj9" event={"ID":"b4177ef3-39b7-4328-9299-d1acac07b82d","Type":"ContainerStarted","Data":"0f09d585475385f35003ad1de7b24076e96d3819cd0bb3c499cee475e178a2fc"}
Jul 27 17:33:25.334433 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:25.334405    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:33:25.370206 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:25.369821    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zfdj9" podStartSLOduration=251.79468077 podStartE2EDuration="4m13.369802879s" totalImagesPullingTime="1.575122109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:25.369334584 +0000 UTC m=+287.414072667" watchObservedRunningTime="2026-07-27 17:33:25.369802879 +0000 UTC m=+287.414540961"
Jul 27 17:33:26.342216 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:26.342178    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx" event={"ID":"35036ce7-166a-4289-b4c2-73b08e3c282e","Type":"ContainerStarted","Data":"4f2b4b687c7114ed2f9a91a2a339b4e74e894eef5181ae5e6d57f0dd80559be2"}
Jul 27 17:33:26.342708 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:26.342258    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx" event={"ID":"35036ce7-166a-4289-b4c2-73b08e3c282e","Type":"ContainerStarted","Data":"9bd9cc210b23838d243ee8ecf7e3020d77057b4ffdad879f9558a4a2cc72bf5a"}
Jul 27 17:33:26.342708 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:26.342298    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx" event={"ID":"35036ce7-166a-4289-b4c2-73b08e3c282e","Type":"ContainerStarted","Data":"5fa536f50f9e83ce1b7e27d5a0c22252e766c3ad1d2c895dd9927c2c2eb337b1"}
Jul 27 17:33:27.347224 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:27.347189    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx" event={"ID":"35036ce7-166a-4289-b4c2-73b08e3c282e","Type":"ContainerStarted","Data":"68ca5a1cb33bfeec5e4a1cb8769cbf4d5b7978bd812563b830c7b31a0c4080d6"}
Jul 27 17:33:27.347656 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:27.347287    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx" event={"ID":"35036ce7-166a-4289-b4c2-73b08e3c282e","Type":"ContainerStarted","Data":"33b2017fda115cf952c1e4fbe41a6e95a26283684e753d24c5700f8d43bc393c"}
Jul 27 17:33:27.347656 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:27.347339    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:27.347656 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:27.347366    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx" event={"ID":"35036ce7-166a-4289-b4c2-73b08e3c282e","Type":"ContainerStarted","Data":"98472747db2a48ca5b7bf6c05d07fdf28359c2ccc7a37867b761964743a019d6"}
Jul 27 17:33:27.370909 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:27.370863    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx" podStartSLOduration=1.982008802 podStartE2EDuration="5.370848713s" totalImagesPullingTime="3.388839911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:33:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:27.368778331 +0000 UTC m=+289.413516412" watchObservedRunningTime="2026-07-27 17:33:27.370848713 +0000 UTC m=+289.415586816"
Jul 27 17:33:29.993093 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:29.993054    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p"
Jul 27 17:33:29.993474 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:29.993428    2571 scope.go:117] "RemoveContainer" containerID="7af5e33a6c8950d1ecbbc8eaa1d3bf635b1778f725cf501c22cec642d0c02720"
Jul 27 17:33:29.993627 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:33:29.993606    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p_open-cluster-management-agent-addon(d7399449-a96d-4790-94f8-88b4cb87533f)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p" podUID="d7399449-a96d-4790-94f8-88b4cb87533f"
Jul 27 17:33:29.998268 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:29.998238    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm" podUID="3e362be5-ec40-471a-8465-48cdf8f30eda" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 27 17:33:31.395257 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:33:31.395202    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vtptl" podUID="50fb06f3-ab71-43cc-aa9c-58ba1b7651d6"
Jul 27 17:33:33.356127 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:33.356097    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f67999996-4qfvx"
Jul 27 17:33:35.344830 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:35.344801    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zfdj9"
Jul 27 17:33:38.259248 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:38.259223    2571 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 17:33:39.998958 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:39.998918    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm" podUID="3e362be5-ec40-471a-8465-48cdf8f30eda" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 27 17:33:40.005733 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:39.998993    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm"
Jul 27 17:33:40.005733 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:39.999578    2571 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"aef40b40b9294ca96e862e8065d977b94ba057e83b49fe128bcc8cc14b69f213"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 27 17:33:40.005733 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:39.999627    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm" podUID="3e362be5-ec40-471a-8465-48cdf8f30eda" containerName="service-proxy" containerID="cri-o://aef40b40b9294ca96e862e8065d977b94ba057e83b49fe128bcc8cc14b69f213" gracePeriod=30
Jul 27 17:33:40.396885 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:40.396787    2571 generic.go:356] "Generic (PLEG): container finished" podID="3e362be5-ec40-471a-8465-48cdf8f30eda" containerID="aef40b40b9294ca96e862e8065d977b94ba057e83b49fe128bcc8cc14b69f213" exitCode=2
Jul 27 17:33:40.399644 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:40.399609    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm" event={"ID":"3e362be5-ec40-471a-8465-48cdf8f30eda","Type":"ContainerDied","Data":"aef40b40b9294ca96e862e8065d977b94ba057e83b49fe128bcc8cc14b69f213"}
Jul 27 17:33:40.399990 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:40.399649    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85db9948db-xhzxm" event={"ID":"3e362be5-ec40-471a-8465-48cdf8f30eda","Type":"ContainerStarted","Data":"5d7b2e84e68a3299db406e75de3a6120c2696fba3508d2efba54d251580f2937"}
Jul 27 17:33:43.392906 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:43.392876    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:33:44.393717 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:44.393682    2571 scope.go:117] "RemoveContainer" containerID="7af5e33a6c8950d1ecbbc8eaa1d3bf635b1778f725cf501c22cec642d0c02720"
Jul 27 17:33:45.414399 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:33:45.414369    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8f47cf66-bgk8p" event={"ID":"d7399449-a96d-4790-94f8-88b4cb87533f","Type":"ContainerStarted","Data":"ca42b782822d9a64a898bd6a08b41e1842e7d12071cd9c93ea7328998d3aa0a8"}
Jul 27 17:34:52.285792 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:34:52.285742    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:34:52.288010 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:34:52.287987    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50fb06f3-ab71-43cc-aa9c-58ba1b7651d6-metrics-certs\") pod \"network-metrics-daemon-vtptl\" (UID: \"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6\") " pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:34:52.396409 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:34:52.396382    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qwvcb\""
Jul 27 17:34:52.403850 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:34:52.403830    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtptl"
Jul 27 17:34:52.524964 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:34:52.524937    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vtptl"]
Jul 27 17:34:52.528193 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:34:52.528169    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fb06f3_ab71_43cc_aa9c_58ba1b7651d6.slice/crio-c1fc59b1515dc7fe4fba511532e468a2234303106cba35255d3aed4963c61568 WatchSource:0}: Error finding container c1fc59b1515dc7fe4fba511532e468a2234303106cba35255d3aed4963c61568: Status 404 returned error can't find the container with id c1fc59b1515dc7fe4fba511532e468a2234303106cba35255d3aed4963c61568
Jul 27 17:34:52.530176 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:34:52.530160    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:34:52.640402 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:34:52.640371    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vtptl" event={"ID":"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6","Type":"ContainerStarted","Data":"c1fc59b1515dc7fe4fba511532e468a2234303106cba35255d3aed4963c61568"}
Jul 27 17:34:54.648908 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:34:54.648881    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vtptl" event={"ID":"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6","Type":"ContainerStarted","Data":"50e21433280697886f119694af171158d5c1e34a5981864600c99742e5492ba2"}
Jul 27 17:34:54.649230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:34:54.648941    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vtptl" event={"ID":"50fb06f3-ab71-43cc-aa9c-58ba1b7651d6","Type":"ContainerStarted","Data":"d5c89d5f7f1af9edb17228f7c79e8fd363c4e1cfe10c027a31a622c8d0a5ab08"}
Jul 27 17:34:54.665711 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:34:54.665670    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vtptl" podStartSLOduration=374.725509927 podStartE2EDuration="6m16.665655558s" totalImagesPullingTime="1.940145631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:34:54.664023893 +0000 UTC m=+376.708761974" watchObservedRunningTime="2026-07-27 17:34:54.665655558 +0000 UTC m=+376.710393640"
Jul 27 17:36:43.413718 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.413676    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"]
Jul 27 17:36:43.417001 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.416979    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"
Jul 27 17:36:43.420127 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.420104    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 17:36:43.420230 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.420202    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xhmts\""
Jul 27 17:36:43.421254 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.421236    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 17:36:43.430426 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.430405    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fkbp\" (UniqueName: \"kubernetes.io/projected/46596894-8c50-45f1-a479-a3b26ca50667-kube-api-access-9fkbp\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz\" (UID: \"46596894-8c50-45f1-a479-a3b26ca50667\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"
Jul 27 17:36:43.430538 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.430434    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46596894-8c50-45f1-a479-a3b26ca50667-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz\" (UID: \"46596894-8c50-45f1-a479-a3b26ca50667\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"
Jul 27 17:36:43.430538 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.430453    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46596894-8c50-45f1-a479-a3b26ca50667-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz\" (UID: \"46596894-8c50-45f1-a479-a3b26ca50667\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"
Jul 27 17:36:43.434485 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.434462    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"]
Jul 27 17:36:43.531561 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.531523    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkbp\" (UniqueName: \"kubernetes.io/projected/46596894-8c50-45f1-a479-a3b26ca50667-kube-api-access-9fkbp\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz\" (UID: \"46596894-8c50-45f1-a479-a3b26ca50667\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"
Jul 27 17:36:43.531749 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.531572    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46596894-8c50-45f1-a479-a3b26ca50667-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz\" (UID: \"46596894-8c50-45f1-a479-a3b26ca50667\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"
Jul 27 17:36:43.531749 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.531593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46596894-8c50-45f1-a479-a3b26ca50667-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz\" (UID: \"46596894-8c50-45f1-a479-a3b26ca50667\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"
Jul 27 17:36:43.531992 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.531975    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46596894-8c50-45f1-a479-a3b26ca50667-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz\" (UID: \"46596894-8c50-45f1-a479-a3b26ca50667\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"
Jul 27 17:36:43.532026 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.531988    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46596894-8c50-45f1-a479-a3b26ca50667-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz\" (UID: \"46596894-8c50-45f1-a479-a3b26ca50667\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"
Jul 27 17:36:43.539950 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.539925    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkbp\" (UniqueName: \"kubernetes.io/projected/46596894-8c50-45f1-a479-a3b26ca50667-kube-api-access-9fkbp\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz\" (UID: \"46596894-8c50-45f1-a479-a3b26ca50667\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"
Jul 27 17:36:43.725624 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.725593    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"
Jul 27 17:36:43.848819 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:43.848788    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"]
Jul 27 17:36:43.852058 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:36:43.852033    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46596894_8c50_45f1_a479_a3b26ca50667.slice/crio-4dc9666cdd7fe539f0a1fab9933b7f0492f602c26404555fc45f922c31bb041b WatchSource:0}: Error finding container 4dc9666cdd7fe539f0a1fab9933b7f0492f602c26404555fc45f922c31bb041b: Status 404 returned error can't find the container with id 4dc9666cdd7fe539f0a1fab9933b7f0492f602c26404555fc45f922c31bb041b
Jul 27 17:36:44.020250 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:44.020164    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz" event={"ID":"46596894-8c50-45f1-a479-a3b26ca50667","Type":"ContainerStarted","Data":"4dc9666cdd7fe539f0a1fab9933b7f0492f602c26404555fc45f922c31bb041b"}
Jul 27 17:36:51.050463 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:51.050425    2571 generic.go:356] "Generic (PLEG): container finished" podID="46596894-8c50-45f1-a479-a3b26ca50667" containerID="909bddbaf5211d8a1054502c3fab5963adb798bbe6cee9e9bd60eb345532c612" exitCode=0
Jul 27 17:36:51.050878 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:51.050472    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz" event={"ID":"46596894-8c50-45f1-a479-a3b26ca50667","Type":"ContainerDied","Data":"909bddbaf5211d8a1054502c3fab5963adb798bbe6cee9e9bd60eb345532c612"}
Jul 27 17:36:54.063086 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:54.063051    2571 generic.go:356] "Generic (PLEG): container finished" podID="46596894-8c50-45f1-a479-a3b26ca50667" containerID="025293f0807099ecbf26602b506b703d484a38099d542beda767f136bd2a2f87" exitCode=0
Jul 27 17:36:54.063479 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:36:54.063120    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz" event={"ID":"46596894-8c50-45f1-a479-a3b26ca50667","Type":"ContainerDied","Data":"025293f0807099ecbf26602b506b703d484a38099d542beda767f136bd2a2f87"}
Jul 27 17:37:01.092189 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:01.092155    2571 generic.go:356] "Generic (PLEG): container finished" podID="46596894-8c50-45f1-a479-a3b26ca50667" containerID="e5897c6e365f1596863741cd3d5f9468806410b4ff715011fb44598575bfbb4f" exitCode=0
Jul 27 17:37:01.092587 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:01.092203    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz" event={"ID":"46596894-8c50-45f1-a479-a3b26ca50667","Type":"ContainerDied","Data":"e5897c6e365f1596863741cd3d5f9468806410b4ff715011fb44598575bfbb4f"}
Jul 27 17:37:02.218251 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:02.218223    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"
Jul 27 17:37:02.262560 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:02.262524    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46596894-8c50-45f1-a479-a3b26ca50667-bundle\") pod \"46596894-8c50-45f1-a479-a3b26ca50667\" (UID: \"46596894-8c50-45f1-a479-a3b26ca50667\") "
Jul 27 17:37:02.262706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:02.262592    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46596894-8c50-45f1-a479-a3b26ca50667-util\") pod \"46596894-8c50-45f1-a479-a3b26ca50667\" (UID: \"46596894-8c50-45f1-a479-a3b26ca50667\") "
Jul 27 17:37:02.262706 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:02.262628    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fkbp\" (UniqueName: \"kubernetes.io/projected/46596894-8c50-45f1-a479-a3b26ca50667-kube-api-access-9fkbp\") pod \"46596894-8c50-45f1-a479-a3b26ca50667\" (UID: \"46596894-8c50-45f1-a479-a3b26ca50667\") "
Jul 27 17:37:02.263269 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:02.263239    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46596894-8c50-45f1-a479-a3b26ca50667-bundle" (OuterVolumeSpecName: "bundle") pod "46596894-8c50-45f1-a479-a3b26ca50667" (UID: "46596894-8c50-45f1-a479-a3b26ca50667"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:37:02.264944 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:02.264912    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46596894-8c50-45f1-a479-a3b26ca50667-kube-api-access-9fkbp" (OuterVolumeSpecName: "kube-api-access-9fkbp") pod "46596894-8c50-45f1-a479-a3b26ca50667" (UID: "46596894-8c50-45f1-a479-a3b26ca50667"). InnerVolumeSpecName "kube-api-access-9fkbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:37:02.266577 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:02.266548    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46596894-8c50-45f1-a479-a3b26ca50667-util" (OuterVolumeSpecName: "util") pod "46596894-8c50-45f1-a479-a3b26ca50667" (UID: "46596894-8c50-45f1-a479-a3b26ca50667"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:37:02.363197 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:02.363121    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46596894-8c50-45f1-a479-a3b26ca50667-bundle\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:37:02.363197 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:02.363145    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46596894-8c50-45f1-a479-a3b26ca50667-util\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:37:02.363197 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:02.363155    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fkbp\" (UniqueName: \"kubernetes.io/projected/46596894-8c50-45f1-a479-a3b26ca50667-kube-api-access-9fkbp\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:37:03.102776 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:03.102738    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz" event={"ID":"46596894-8c50-45f1-a479-a3b26ca50667","Type":"ContainerDied","Data":"4dc9666cdd7fe539f0a1fab9933b7f0492f602c26404555fc45f922c31bb041b"}
Jul 27 17:37:03.102776 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:03.102774    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b59jwrz"
Jul 27 17:37:03.102976 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:37:03.102790    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dc9666cdd7fe539f0a1fab9933b7f0492f602c26404555fc45f922c31bb041b"
Jul 27 17:38:26.275898 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.275864    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-qz9gv"]
Jul 27 17:38:26.276365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.276145    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46596894-8c50-45f1-a479-a3b26ca50667" containerName="pull"
Jul 27 17:38:26.276365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.276156    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="46596894-8c50-45f1-a479-a3b26ca50667" containerName="pull"
Jul 27 17:38:26.276365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.276172    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46596894-8c50-45f1-a479-a3b26ca50667" containerName="util"
Jul 27 17:38:26.276365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.276177    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="46596894-8c50-45f1-a479-a3b26ca50667" containerName="util"
Jul 27 17:38:26.276365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.276189    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46596894-8c50-45f1-a479-a3b26ca50667" containerName="extract"
Jul 27 17:38:26.276365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.276195    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="46596894-8c50-45f1-a479-a3b26ca50667" containerName="extract"
Jul 27 17:38:26.276365 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.276235    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="46596894-8c50-45f1-a479-a3b26ca50667" containerName="extract"
Jul 27 17:38:26.279032 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.279015    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-qz9gv"
Jul 27 17:38:26.281553 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.281530    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 27 17:38:26.281686 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.281601    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 27 17:38:26.282717 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.282681    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-x4gd8\""
Jul 27 17:38:26.282819 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.282694    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 27 17:38:26.287815 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.287789    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-qz9gv"]
Jul 27 17:38:26.327124 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.327096    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-q5r7l"]
Jul 27 17:38:26.330078 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.330059    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-q5r7l"
Jul 27 17:38:26.332528 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.332511    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 27 17:38:26.332599 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.332534    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jrmwf\""
Jul 27 17:38:26.338422 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.338402    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-q5r7l"]
Jul 27 17:38:26.369090 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.369065    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c57478f8-0603-4e94-a5c2-c06a1babde8d-cert\") pod \"kserve-controller-manager-ff96d5959-qz9gv\" (UID: \"c57478f8-0603-4e94-a5c2-c06a1babde8d\") " pod="kserve/kserve-controller-manager-ff96d5959-qz9gv"
Jul 27 17:38:26.369182 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.369099    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dbe2bbc7-a708-4d74-9963-c609e16dc4cc-data\") pod \"seaweedfs-64568bcd49-q5r7l\" (UID: \"dbe2bbc7-a708-4d74-9963-c609e16dc4cc\") " pod="kserve/seaweedfs-64568bcd49-q5r7l"
Jul 27 17:38:26.369182 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.369124    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbxrk\" (UniqueName: \"kubernetes.io/projected/c57478f8-0603-4e94-a5c2-c06a1babde8d-kube-api-access-tbxrk\") pod \"kserve-controller-manager-ff96d5959-qz9gv\" (UID: \"c57478f8-0603-4e94-a5c2-c06a1babde8d\") " pod="kserve/kserve-controller-manager-ff96d5959-qz9gv"
Jul 27 17:38:26.369267 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.369205    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cfxc\" (UniqueName: \"kubernetes.io/projected/dbe2bbc7-a708-4d74-9963-c609e16dc4cc-kube-api-access-6cfxc\") pod \"seaweedfs-64568bcd49-q5r7l\" (UID: \"dbe2bbc7-a708-4d74-9963-c609e16dc4cc\") " pod="kserve/seaweedfs-64568bcd49-q5r7l"
Jul 27 17:38:26.470073 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.470046    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cfxc\" (UniqueName: \"kubernetes.io/projected/dbe2bbc7-a708-4d74-9963-c609e16dc4cc-kube-api-access-6cfxc\") pod \"seaweedfs-64568bcd49-q5r7l\" (UID: \"dbe2bbc7-a708-4d74-9963-c609e16dc4cc\") " pod="kserve/seaweedfs-64568bcd49-q5r7l"
Jul 27 17:38:26.470204 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.470081    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c57478f8-0603-4e94-a5c2-c06a1babde8d-cert\") pod \"kserve-controller-manager-ff96d5959-qz9gv\" (UID: \"c57478f8-0603-4e94-a5c2-c06a1babde8d\") " pod="kserve/kserve-controller-manager-ff96d5959-qz9gv"
Jul 27 17:38:26.470204 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.470104    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dbe2bbc7-a708-4d74-9963-c609e16dc4cc-data\") pod \"seaweedfs-64568bcd49-q5r7l\" (UID: \"dbe2bbc7-a708-4d74-9963-c609e16dc4cc\") " pod="kserve/seaweedfs-64568bcd49-q5r7l"
Jul 27 17:38:26.470204 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.470129    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxrk\" (UniqueName: \"kubernetes.io/projected/c57478f8-0603-4e94-a5c2-c06a1babde8d-kube-api-access-tbxrk\") pod \"kserve-controller-manager-ff96d5959-qz9gv\" (UID: \"c57478f8-0603-4e94-a5c2-c06a1babde8d\") " pod="kserve/kserve-controller-manager-ff96d5959-qz9gv"
Jul 27 17:38:26.470204 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:38:26.470199    2571 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 27 17:38:26.470384 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:38:26.470259    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c57478f8-0603-4e94-a5c2-c06a1babde8d-cert podName:c57478f8-0603-4e94-a5c2-c06a1babde8d nodeName:}" failed. No retries permitted until 2026-07-27 17:38:26.970242274 +0000 UTC m=+589.014980336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c57478f8-0603-4e94-a5c2-c06a1babde8d-cert") pod "kserve-controller-manager-ff96d5959-qz9gv" (UID: "c57478f8-0603-4e94-a5c2-c06a1babde8d") : secret "kserve-webhook-server-cert" not found
Jul 27 17:38:26.470435 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.470399    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dbe2bbc7-a708-4d74-9963-c609e16dc4cc-data\") pod \"seaweedfs-64568bcd49-q5r7l\" (UID: \"dbe2bbc7-a708-4d74-9963-c609e16dc4cc\") " pod="kserve/seaweedfs-64568bcd49-q5r7l"
Jul 27 17:38:26.478758 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.478733    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cfxc\" (UniqueName: \"kubernetes.io/projected/dbe2bbc7-a708-4d74-9963-c609e16dc4cc-kube-api-access-6cfxc\") pod \"seaweedfs-64568bcd49-q5r7l\" (UID: \"dbe2bbc7-a708-4d74-9963-c609e16dc4cc\") " pod="kserve/seaweedfs-64568bcd49-q5r7l"
Jul 27 17:38:26.479405 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.479386    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxrk\" (UniqueName: \"kubernetes.io/projected/c57478f8-0603-4e94-a5c2-c06a1babde8d-kube-api-access-tbxrk\") pod \"kserve-controller-manager-ff96d5959-qz9gv\" (UID: \"c57478f8-0603-4e94-a5c2-c06a1babde8d\") " pod="kserve/kserve-controller-manager-ff96d5959-qz9gv"
Jul 27 17:38:26.639207 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.639132    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-q5r7l"
Jul 27 17:38:26.761735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.761702    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-q5r7l"]
Jul 27 17:38:26.764845 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:38:26.764817    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe2bbc7_a708_4d74_9963_c609e16dc4cc.slice/crio-0b8b21e0b897b6616943ff88068f8d56b35a6beb8931f6389feed7b5c47d1ba3 WatchSource:0}: Error finding container 0b8b21e0b897b6616943ff88068f8d56b35a6beb8931f6389feed7b5c47d1ba3: Status 404 returned error can't find the container with id 0b8b21e0b897b6616943ff88068f8d56b35a6beb8931f6389feed7b5c47d1ba3
Jul 27 17:38:26.973338 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.973301    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c57478f8-0603-4e94-a5c2-c06a1babde8d-cert\") pod \"kserve-controller-manager-ff96d5959-qz9gv\" (UID: \"c57478f8-0603-4e94-a5c2-c06a1babde8d\") " pod="kserve/kserve-controller-manager-ff96d5959-qz9gv"
Jul 27 17:38:26.975647 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:26.975627    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c57478f8-0603-4e94-a5c2-c06a1babde8d-cert\") pod \"kserve-controller-manager-ff96d5959-qz9gv\" (UID: \"c57478f8-0603-4e94-a5c2-c06a1babde8d\") " pod="kserve/kserve-controller-manager-ff96d5959-qz9gv"
Jul 27 17:38:27.189153 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:27.189120    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-qz9gv"
Jul 27 17:38:27.315019 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:27.314953    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-qz9gv"]
Jul 27 17:38:27.317318 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:38:27.317289    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57478f8_0603_4e94_a5c2_c06a1babde8d.slice/crio-e7e07a80c744afdea9fa188a07e4bd9d6c8bbe1e7ba2fa426acf923f423fe92f WatchSource:0}: Error finding container e7e07a80c744afdea9fa188a07e4bd9d6c8bbe1e7ba2fa426acf923f423fe92f: Status 404 returned error can't find the container with id e7e07a80c744afdea9fa188a07e4bd9d6c8bbe1e7ba2fa426acf923f423fe92f
Jul 27 17:38:27.389571 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:27.389536    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-qz9gv" event={"ID":"c57478f8-0603-4e94-a5c2-c06a1babde8d","Type":"ContainerStarted","Data":"e7e07a80c744afdea9fa188a07e4bd9d6c8bbe1e7ba2fa426acf923f423fe92f"}
Jul 27 17:38:27.391049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:27.391007    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-q5r7l" event={"ID":"dbe2bbc7-a708-4d74-9963-c609e16dc4cc","Type":"ContainerStarted","Data":"0b8b21e0b897b6616943ff88068f8d56b35a6beb8931f6389feed7b5c47d1ba3"}
Jul 27 17:38:31.414886 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:31.414791    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-qz9gv" event={"ID":"c57478f8-0603-4e94-a5c2-c06a1babde8d","Type":"ContainerStarted","Data":"1f1016d68f04a30ec5a393f9eff98863e3df8b3fc1a1c5b13e2bf60d4b67211f"}
Jul 27 17:38:31.415308 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:31.414882    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-qz9gv"
Jul 27 17:38:31.416197 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:31.416174    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-q5r7l" event={"ID":"dbe2bbc7-a708-4d74-9963-c609e16dc4cc","Type":"ContainerStarted","Data":"6d3a4dbef90321b9ec4b4711db8083f5b9d6232602eda49ffecf81e9c4ead3c5"}
Jul 27 17:38:31.416312 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:31.416228    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-q5r7l"
Jul 27 17:38:31.430743 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:31.430705    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-qz9gv" podStartSLOduration=1.799810237 podStartE2EDuration="5.430693717s" totalImagesPullingTime="3.63088348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:38:31.430597104 +0000 UTC m=+593.475335185" watchObservedRunningTime="2026-07-27 17:38:31.430693717 +0000 UTC m=+593.475431798"
Jul 27 17:38:31.445910 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:31.445870    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-q5r7l" podStartSLOduration=1.212073338 podStartE2EDuration="5.445859411s" totalImagesPullingTime="4.233786073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:38:31.444904926 +0000 UTC m=+593.489643006" watchObservedRunningTime="2026-07-27 17:38:31.445859411 +0000 UTC m=+593.490597492"
Jul 27 17:38:37.422430 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:38:37.422395    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-q5r7l"
Jul 27 17:39:02.425019 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:02.424949    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-qz9gv"
Jul 27 17:39:09.613993 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.613964    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"]
Jul 27 17:39:09.617046 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.617031    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"
Jul 27 17:39:09.621108 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.621083    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 17:39:09.621251 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.621086    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xhmts\""
Jul 27 17:39:09.622282 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.622262    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 17:39:09.627737 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.627716    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"]
Jul 27 17:39:09.783747 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.783702    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqvnh\" (UniqueName: \"kubernetes.io/projected/4e396248-4317-46b1-a174-e5a40e592243-kube-api-access-pqvnh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk\" (UID: \"4e396248-4317-46b1-a174-e5a40e592243\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"
Jul 27 17:39:09.783747 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.783754    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e396248-4317-46b1-a174-e5a40e592243-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk\" (UID: \"4e396248-4317-46b1-a174-e5a40e592243\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"
Jul 27 17:39:09.784002 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.783863    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e396248-4317-46b1-a174-e5a40e592243-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk\" (UID: \"4e396248-4317-46b1-a174-e5a40e592243\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"
Jul 27 17:39:09.885077 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.884978    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e396248-4317-46b1-a174-e5a40e592243-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk\" (UID: \"4e396248-4317-46b1-a174-e5a40e592243\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"
Jul 27 17:39:09.885231 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.885140    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e396248-4317-46b1-a174-e5a40e592243-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk\" (UID: \"4e396248-4317-46b1-a174-e5a40e592243\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"
Jul 27 17:39:09.885231 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.885196    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqvnh\" (UniqueName: \"kubernetes.io/projected/4e396248-4317-46b1-a174-e5a40e592243-kube-api-access-pqvnh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk\" (UID: \"4e396248-4317-46b1-a174-e5a40e592243\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"
Jul 27 17:39:09.885439 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.885416    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e396248-4317-46b1-a174-e5a40e592243-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk\" (UID: \"4e396248-4317-46b1-a174-e5a40e592243\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"
Jul 27 17:39:09.885564 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.885476    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e396248-4317-46b1-a174-e5a40e592243-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk\" (UID: \"4e396248-4317-46b1-a174-e5a40e592243\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"
Jul 27 17:39:09.893219 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.893201    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqvnh\" (UniqueName: \"kubernetes.io/projected/4e396248-4317-46b1-a174-e5a40e592243-kube-api-access-pqvnh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk\" (UID: \"4e396248-4317-46b1-a174-e5a40e592243\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"
Jul 27 17:39:09.925676 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:09.925657    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"
Jul 27 17:39:10.048295 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:10.048264    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"]
Jul 27 17:39:10.052223 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:39:10.052192    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e396248_4317_46b1_a174_e5a40e592243.slice/crio-a968e7c54b12967bb69a478ed56e3d88286738d5d7c5fcf9d6c5c25c06a31f60 WatchSource:0}: Error finding container a968e7c54b12967bb69a478ed56e3d88286738d5d7c5fcf9d6c5c25c06a31f60: Status 404 returned error can't find the container with id a968e7c54b12967bb69a478ed56e3d88286738d5d7c5fcf9d6c5c25c06a31f60
Jul 27 17:39:10.556002 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:10.555964    2571 generic.go:356] "Generic (PLEG): container finished" podID="4e396248-4317-46b1-a174-e5a40e592243" containerID="5e2d26f367264e4370a532b8f4b2ed78ca34a18554b175fcb262672cb815290f" exitCode=0
Jul 27 17:39:10.556166 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:10.556035    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk" event={"ID":"4e396248-4317-46b1-a174-e5a40e592243","Type":"ContainerDied","Data":"5e2d26f367264e4370a532b8f4b2ed78ca34a18554b175fcb262672cb815290f"}
Jul 27 17:39:10.556166 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:10.556078    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk" event={"ID":"4e396248-4317-46b1-a174-e5a40e592243","Type":"ContainerStarted","Data":"a968e7c54b12967bb69a478ed56e3d88286738d5d7c5fcf9d6c5c25c06a31f60"}
Jul 27 17:39:13.568478 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:13.568437    2571 generic.go:356] "Generic (PLEG): container finished" podID="4e396248-4317-46b1-a174-e5a40e592243" containerID="60b684a07a8f46ecf966b734d902a03c687b2a891bc7c61aa642cad4edcb80f2" exitCode=0
Jul 27 17:39:13.568847 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:13.568487    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk" event={"ID":"4e396248-4317-46b1-a174-e5a40e592243","Type":"ContainerDied","Data":"60b684a07a8f46ecf966b734d902a03c687b2a891bc7c61aa642cad4edcb80f2"}
Jul 27 17:39:14.574479 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:14.574442    2571 generic.go:356] "Generic (PLEG): container finished" podID="4e396248-4317-46b1-a174-e5a40e592243" containerID="fd2daff3b438c4f2ca8f0cb56cf06c1b849ec928693e0a4b5eb6b37b865fe088" exitCode=0
Jul 27 17:39:14.574964 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:14.574532    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk" event={"ID":"4e396248-4317-46b1-a174-e5a40e592243","Type":"ContainerDied","Data":"fd2daff3b438c4f2ca8f0cb56cf06c1b849ec928693e0a4b5eb6b37b865fe088"}
Jul 27 17:39:15.705685 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:15.705661    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"
Jul 27 17:39:15.832454 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:15.832374    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqvnh\" (UniqueName: \"kubernetes.io/projected/4e396248-4317-46b1-a174-e5a40e592243-kube-api-access-pqvnh\") pod \"4e396248-4317-46b1-a174-e5a40e592243\" (UID: \"4e396248-4317-46b1-a174-e5a40e592243\") "
Jul 27 17:39:15.832454 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:15.832407    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e396248-4317-46b1-a174-e5a40e592243-util\") pod \"4e396248-4317-46b1-a174-e5a40e592243\" (UID: \"4e396248-4317-46b1-a174-e5a40e592243\") "
Jul 27 17:39:15.832454 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:15.832435    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e396248-4317-46b1-a174-e5a40e592243-bundle\") pod \"4e396248-4317-46b1-a174-e5a40e592243\" (UID: \"4e396248-4317-46b1-a174-e5a40e592243\") "
Jul 27 17:39:15.833142 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:15.833114    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e396248-4317-46b1-a174-e5a40e592243-bundle" (OuterVolumeSpecName: "bundle") pod "4e396248-4317-46b1-a174-e5a40e592243" (UID: "4e396248-4317-46b1-a174-e5a40e592243"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:39:15.834454 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:15.834427    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e396248-4317-46b1-a174-e5a40e592243-kube-api-access-pqvnh" (OuterVolumeSpecName: "kube-api-access-pqvnh") pod "4e396248-4317-46b1-a174-e5a40e592243" (UID: "4e396248-4317-46b1-a174-e5a40e592243"). InnerVolumeSpecName "kube-api-access-pqvnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:39:15.837282 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:15.837244    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e396248-4317-46b1-a174-e5a40e592243-util" (OuterVolumeSpecName: "util") pod "4e396248-4317-46b1-a174-e5a40e592243" (UID: "4e396248-4317-46b1-a174-e5a40e592243"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:39:15.932934 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:15.932888    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e396248-4317-46b1-a174-e5a40e592243-bundle\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:39:15.932934 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:15.932928    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqvnh\" (UniqueName: \"kubernetes.io/projected/4e396248-4317-46b1-a174-e5a40e592243-kube-api-access-pqvnh\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:39:15.932934 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:15.932942    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e396248-4317-46b1-a174-e5a40e592243-util\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:39:16.585260 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:16.585233    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk"
Jul 27 17:39:16.585452 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:16.585232    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w9brk" event={"ID":"4e396248-4317-46b1-a174-e5a40e592243","Type":"ContainerDied","Data":"a968e7c54b12967bb69a478ed56e3d88286738d5d7c5fcf9d6c5c25c06a31f60"}
Jul 27 17:39:16.585452 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:16.585352    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a968e7c54b12967bb69a478ed56e3d88286738d5d7c5fcf9d6c5c25c06a31f60"
Jul 27 17:39:22.566117 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.566083    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld"]
Jul 27 17:39:22.566524 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.566357    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e396248-4317-46b1-a174-e5a40e592243" containerName="util"
Jul 27 17:39:22.566524 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.566368    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e396248-4317-46b1-a174-e5a40e592243" containerName="util"
Jul 27 17:39:22.566524 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.566385    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e396248-4317-46b1-a174-e5a40e592243" containerName="extract"
Jul 27 17:39:22.566524 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.566394    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e396248-4317-46b1-a174-e5a40e592243" containerName="extract"
Jul 27 17:39:22.566524 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.566405    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e396248-4317-46b1-a174-e5a40e592243" containerName="pull"
Jul 27 17:39:22.566524 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.566410    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e396248-4317-46b1-a174-e5a40e592243" containerName="pull"
Jul 27 17:39:22.566524 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.566448    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e396248-4317-46b1-a174-e5a40e592243" containerName="extract"
Jul 27 17:39:22.570321 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.570301    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld"
Jul 27 17:39:22.572940 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.572919    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:39:22.573047 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.572922    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 27 17:39:22.573047 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.572923    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-smjrp\""
Jul 27 17:39:22.582727 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.582707    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld"]
Jul 27 17:39:22.681628 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.681579    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9rjn\" (UniqueName: \"kubernetes.io/projected/eba232cb-21b2-48cc-87d6-619cc67a4a7c-kube-api-access-d9rjn\") pod \"cert-manager-operator-controller-manager-684f79c9b9-2gbld\" (UID: \"eba232cb-21b2-48cc-87d6-619cc67a4a7c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld"
Jul 27 17:39:22.681817 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.681648    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eba232cb-21b2-48cc-87d6-619cc67a4a7c-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-2gbld\" (UID: \"eba232cb-21b2-48cc-87d6-619cc67a4a7c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld"
Jul 27 17:39:22.782822 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.782772    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rjn\" (UniqueName: \"kubernetes.io/projected/eba232cb-21b2-48cc-87d6-619cc67a4a7c-kube-api-access-d9rjn\") pod \"cert-manager-operator-controller-manager-684f79c9b9-2gbld\" (UID: \"eba232cb-21b2-48cc-87d6-619cc67a4a7c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld"
Jul 27 17:39:22.782990 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.782862    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eba232cb-21b2-48cc-87d6-619cc67a4a7c-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-2gbld\" (UID: \"eba232cb-21b2-48cc-87d6-619cc67a4a7c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld"
Jul 27 17:39:22.783257 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.783238    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eba232cb-21b2-48cc-87d6-619cc67a4a7c-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-2gbld\" (UID: \"eba232cb-21b2-48cc-87d6-619cc67a4a7c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld"
Jul 27 17:39:22.791635 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.791602    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9rjn\" (UniqueName: \"kubernetes.io/projected/eba232cb-21b2-48cc-87d6-619cc67a4a7c-kube-api-access-d9rjn\") pod \"cert-manager-operator-controller-manager-684f79c9b9-2gbld\" (UID: \"eba232cb-21b2-48cc-87d6-619cc67a4a7c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld"
Jul 27 17:39:22.879164 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:22.879079    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld"
Jul 27 17:39:23.011960 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:23.011928    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld"]
Jul 27 17:39:23.017810 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:39:23.015368    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba232cb_21b2_48cc_87d6_619cc67a4a7c.slice/crio-1468be62a90c6d345b34ede6ac3535aaf7c260b53fb95517102ad2a0e1a06ed5 WatchSource:0}: Error finding container 1468be62a90c6d345b34ede6ac3535aaf7c260b53fb95517102ad2a0e1a06ed5: Status 404 returned error can't find the container with id 1468be62a90c6d345b34ede6ac3535aaf7c260b53fb95517102ad2a0e1a06ed5
Jul 27 17:39:23.613478 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:23.613441    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld" event={"ID":"eba232cb-21b2-48cc-87d6-619cc67a4a7c","Type":"ContainerStarted","Data":"1468be62a90c6d345b34ede6ac3535aaf7c260b53fb95517102ad2a0e1a06ed5"}
Jul 27 17:39:25.623216 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:25.623171    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld" event={"ID":"eba232cb-21b2-48cc-87d6-619cc67a4a7c","Type":"ContainerStarted","Data":"447fbfd23e83a07d08f211d49a7ac68e858b0379f1995c1f58a896ba07186af3"}
Jul 27 17:39:25.666389 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:25.666330    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-2gbld" podStartSLOduration=1.613892553 podStartE2EDuration="3.666308615s" totalImagesPullingTime="2.052416062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:39:25.66342446 +0000 UTC m=+647.708162545" watchObservedRunningTime="2026-07-27 17:39:25.666308615 +0000 UTC m=+647.711046697"
Jul 27 17:39:29.305631 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.305597    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-sd882"]
Jul 27 17:39:29.308869 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.308851    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sd882"
Jul 27 17:39:29.311527 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.311483    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 27 17:39:29.312612 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.312596    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xtwft\""
Jul 27 17:39:29.312697 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.312598    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 27 17:39:29.318429 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.318407    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-sd882"]
Jul 27 17:39:29.328659 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.328636    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkqrf\" (UniqueName: \"kubernetes.io/projected/2e977b4a-17f7-4466-a77a-58eb9e9c0f0e-kube-api-access-hkqrf\") pod \"cert-manager-cainjector-559cdd8c87-sd882\" (UID: \"2e977b4a-17f7-4466-a77a-58eb9e9c0f0e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sd882"
Jul 27 17:39:29.328739 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.328676    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e977b4a-17f7-4466-a77a-58eb9e9c0f0e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-sd882\" (UID: \"2e977b4a-17f7-4466-a77a-58eb9e9c0f0e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sd882"
Jul 27 17:39:29.429665 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.429637    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkqrf\" (UniqueName: \"kubernetes.io/projected/2e977b4a-17f7-4466-a77a-58eb9e9c0f0e-kube-api-access-hkqrf\") pod \"cert-manager-cainjector-559cdd8c87-sd882\" (UID: \"2e977b4a-17f7-4466-a77a-58eb9e9c0f0e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sd882"
Jul 27 17:39:29.429822 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.429680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e977b4a-17f7-4466-a77a-58eb9e9c0f0e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-sd882\" (UID: \"2e977b4a-17f7-4466-a77a-58eb9e9c0f0e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sd882"
Jul 27 17:39:29.438694 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.438658    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e977b4a-17f7-4466-a77a-58eb9e9c0f0e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-sd882\" (UID: \"2e977b4a-17f7-4466-a77a-58eb9e9c0f0e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sd882"
Jul 27 17:39:29.438887 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.438868    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkqrf\" (UniqueName: \"kubernetes.io/projected/2e977b4a-17f7-4466-a77a-58eb9e9c0f0e-kube-api-access-hkqrf\") pod \"cert-manager-cainjector-559cdd8c87-sd882\" (UID: \"2e977b4a-17f7-4466-a77a-58eb9e9c0f0e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-sd882"
Jul 27 17:39:29.617897 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.617812    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sd882"
Jul 27 17:39:29.743124 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:29.743095    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-sd882"]
Jul 27 17:39:29.746643 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:39:29.746614    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e977b4a_17f7_4466_a77a_58eb9e9c0f0e.slice/crio-5e8d449d0f5dc66d62e8f756960062a29eb423753dc820e23c452568242636e9 WatchSource:0}: Error finding container 5e8d449d0f5dc66d62e8f756960062a29eb423753dc820e23c452568242636e9: Status 404 returned error can't find the container with id 5e8d449d0f5dc66d62e8f756960062a29eb423753dc820e23c452568242636e9
Jul 27 17:39:30.651436 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:30.651400    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sd882" event={"ID":"2e977b4a-17f7-4466-a77a-58eb9e9c0f0e","Type":"ContainerStarted","Data":"5e8d449d0f5dc66d62e8f756960062a29eb423753dc820e23c452568242636e9"}
Jul 27 17:39:32.129676 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.129637    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"]
Jul 27 17:39:32.133247 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.133227    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"
Jul 27 17:39:32.135886 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.135865    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 17:39:32.136985 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.136967    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xhmts\""
Jul 27 17:39:32.137101 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.137018    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 17:39:32.140851 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.140799    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"]
Jul 27 17:39:32.147968 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.147947    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5dc079a-98db-468b-839c-bdcbd133b73b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v\" (UID: \"e5dc079a-98db-468b-839c-bdcbd133b73b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"
Jul 27 17:39:32.148071 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.147979    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qph2c\" (UniqueName: \"kubernetes.io/projected/e5dc079a-98db-468b-839c-bdcbd133b73b-kube-api-access-qph2c\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v\" (UID: \"e5dc079a-98db-468b-839c-bdcbd133b73b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"
Jul 27 17:39:32.148132 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.148071    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5dc079a-98db-468b-839c-bdcbd133b73b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v\" (UID: \"e5dc079a-98db-468b-839c-bdcbd133b73b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"
Jul 27 17:39:32.249200 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.249166    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5dc079a-98db-468b-839c-bdcbd133b73b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v\" (UID: \"e5dc079a-98db-468b-839c-bdcbd133b73b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"
Jul 27 17:39:32.249200 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.249209    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qph2c\" (UniqueName: \"kubernetes.io/projected/e5dc079a-98db-468b-839c-bdcbd133b73b-kube-api-access-qph2c\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v\" (UID: \"e5dc079a-98db-468b-839c-bdcbd133b73b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"
Jul 27 17:39:32.249459 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.249282    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5dc079a-98db-468b-839c-bdcbd133b73b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v\" (UID: \"e5dc079a-98db-468b-839c-bdcbd133b73b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"
Jul 27 17:39:32.249693 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.249664    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5dc079a-98db-468b-839c-bdcbd133b73b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v\" (UID: \"e5dc079a-98db-468b-839c-bdcbd133b73b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"
Jul 27 17:39:32.249827 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.249700    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5dc079a-98db-468b-839c-bdcbd133b73b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v\" (UID: \"e5dc079a-98db-468b-839c-bdcbd133b73b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"
Jul 27 17:39:32.258311 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.258281    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qph2c\" (UniqueName: \"kubernetes.io/projected/e5dc079a-98db-468b-839c-bdcbd133b73b-kube-api-access-qph2c\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v\" (UID: \"e5dc079a-98db-468b-839c-bdcbd133b73b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"
Jul 27 17:39:32.445228 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.445201    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"
Jul 27 17:39:32.576908 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.576874    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"]
Jul 27 17:39:32.579966 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:39:32.579925    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dc079a_98db_468b_839c_bdcbd133b73b.slice/crio-b5eb5ab9566d2d7e143fc575cfb25ec861904a0a6e203ba77a2d6717d51a13c4 WatchSource:0}: Error finding container b5eb5ab9566d2d7e143fc575cfb25ec861904a0a6e203ba77a2d6717d51a13c4: Status 404 returned error can't find the container with id b5eb5ab9566d2d7e143fc575cfb25ec861904a0a6e203ba77a2d6717d51a13c4
Jul 27 17:39:32.661377 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.661343    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sd882" event={"ID":"2e977b4a-17f7-4466-a77a-58eb9e9c0f0e","Type":"ContainerStarted","Data":"b60936cc44eaa6212f9c86537c97599ad7b54c4aeed97baed717e8a169335e44"}
Jul 27 17:39:32.662986 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.662959    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v" event={"ID":"e5dc079a-98db-468b-839c-bdcbd133b73b","Type":"ContainerStarted","Data":"345419f8a01b59a78f7552970b77ec4bcdbc1399c0fee946ef29383aa919bb87"}
Jul 27 17:39:32.663106 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.663012    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v" event={"ID":"e5dc079a-98db-468b-839c-bdcbd133b73b","Type":"ContainerStarted","Data":"b5eb5ab9566d2d7e143fc575cfb25ec861904a0a6e203ba77a2d6717d51a13c4"}
Jul 27 17:39:32.679418 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:32.679373    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-sd882" podStartSLOduration=1.076181084 podStartE2EDuration="3.679354728s" totalImagesPullingTime="2.603173644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:39:32.678235319 +0000 UTC m=+654.722973401" watchObservedRunningTime="2026-07-27 17:39:32.679354728 +0000 UTC m=+654.724092813"
Jul 27 17:39:33.668856 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:33.668821    2571 generic.go:356] "Generic (PLEG): container finished" podID="e5dc079a-98db-468b-839c-bdcbd133b73b" containerID="345419f8a01b59a78f7552970b77ec4bcdbc1399c0fee946ef29383aa919bb87" exitCode=0
Jul 27 17:39:33.669312 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:33.668913    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v" event={"ID":"e5dc079a-98db-468b-839c-bdcbd133b73b","Type":"ContainerDied","Data":"345419f8a01b59a78f7552970b77ec4bcdbc1399c0fee946ef29383aa919bb87"}
Jul 27 17:39:36.682114 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:36.682035    2571 generic.go:356] "Generic (PLEG): container finished" podID="e5dc079a-98db-468b-839c-bdcbd133b73b" containerID="48141792018d263d379ef4765b3e38d5d41412e1edebce02fc9eec3f27e2421f" exitCode=0
Jul 27 17:39:36.682114 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:36.682089    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v" event={"ID":"e5dc079a-98db-468b-839c-bdcbd133b73b","Type":"ContainerDied","Data":"48141792018d263d379ef4765b3e38d5d41412e1edebce02fc9eec3f27e2421f"}
Jul 27 17:39:37.688075 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:37.688040    2571 generic.go:356] "Generic (PLEG): container finished" podID="e5dc079a-98db-468b-839c-bdcbd133b73b" containerID="95e46f74afd274d496d8540941b75fa3c6e9ada37a046b55f283d79a692684c5" exitCode=0
Jul 27 17:39:37.688466 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:37.688094    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v" event={"ID":"e5dc079a-98db-468b-839c-bdcbd133b73b","Type":"ContainerDied","Data":"95e46f74afd274d496d8540941b75fa3c6e9ada37a046b55f283d79a692684c5"}
Jul 27 17:39:38.814873 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:38.814845    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"
Jul 27 17:39:38.893590 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:38.893560    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5dc079a-98db-468b-839c-bdcbd133b73b-util\") pod \"e5dc079a-98db-468b-839c-bdcbd133b73b\" (UID: \"e5dc079a-98db-468b-839c-bdcbd133b73b\") "
Jul 27 17:39:38.893746 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:38.893601    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5dc079a-98db-468b-839c-bdcbd133b73b-bundle\") pod \"e5dc079a-98db-468b-839c-bdcbd133b73b\" (UID: \"e5dc079a-98db-468b-839c-bdcbd133b73b\") "
Jul 27 17:39:38.893746 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:38.893634    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qph2c\" (UniqueName: \"kubernetes.io/projected/e5dc079a-98db-468b-839c-bdcbd133b73b-kube-api-access-qph2c\") pod \"e5dc079a-98db-468b-839c-bdcbd133b73b\" (UID: \"e5dc079a-98db-468b-839c-bdcbd133b73b\") "
Jul 27 17:39:38.894036 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:38.894007    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5dc079a-98db-468b-839c-bdcbd133b73b-bundle" (OuterVolumeSpecName: "bundle") pod "e5dc079a-98db-468b-839c-bdcbd133b73b" (UID: "e5dc079a-98db-468b-839c-bdcbd133b73b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:39:38.895733 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:38.895708    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5dc079a-98db-468b-839c-bdcbd133b73b-kube-api-access-qph2c" (OuterVolumeSpecName: "kube-api-access-qph2c") pod "e5dc079a-98db-468b-839c-bdcbd133b73b" (UID: "e5dc079a-98db-468b-839c-bdcbd133b73b"). InnerVolumeSpecName "kube-api-access-qph2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:39:38.900360 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:38.900317    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5dc079a-98db-468b-839c-bdcbd133b73b-util" (OuterVolumeSpecName: "util") pod "e5dc079a-98db-468b-839c-bdcbd133b73b" (UID: "e5dc079a-98db-468b-839c-bdcbd133b73b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:39:38.994583 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:38.994558    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5dc079a-98db-468b-839c-bdcbd133b73b-util\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:39:38.994583 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:38.994582    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5dc079a-98db-468b-839c-bdcbd133b73b-bundle\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:39:38.994814 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:38.994592    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qph2c\" (UniqueName: \"kubernetes.io/projected/e5dc079a-98db-468b-839c-bdcbd133b73b-kube-api-access-qph2c\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:39:39.698436 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:39.698400    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v" event={"ID":"e5dc079a-98db-468b-839c-bdcbd133b73b","Type":"ContainerDied","Data":"b5eb5ab9566d2d7e143fc575cfb25ec861904a0a6e203ba77a2d6717d51a13c4"}
Jul 27 17:39:39.698620 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:39.698444    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgqq2v"
Jul 27 17:39:39.698620 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:39.698447    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5eb5ab9566d2d7e143fc575cfb25ec861904a0a6e203ba77a2d6717d51a13c4"
Jul 27 17:39:46.642285 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.642254    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc"]
Jul 27 17:39:46.642806 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.642529    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5dc079a-98db-468b-839c-bdcbd133b73b" containerName="util"
Jul 27 17:39:46.642806 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.642540    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5dc079a-98db-468b-839c-bdcbd133b73b" containerName="util"
Jul 27 17:39:46.642806 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.642549    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5dc079a-98db-468b-839c-bdcbd133b73b" containerName="extract"
Jul 27 17:39:46.642806 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.642554    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5dc079a-98db-468b-839c-bdcbd133b73b" containerName="extract"
Jul 27 17:39:46.642806 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.642564    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5dc079a-98db-468b-839c-bdcbd133b73b" containerName="pull"
Jul 27 17:39:46.642806 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.642569    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5dc079a-98db-468b-839c-bdcbd133b73b" containerName="pull"
Jul 27 17:39:46.642806 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.642615    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5dc079a-98db-468b-839c-bdcbd133b73b" containerName="extract"
Jul 27 17:39:46.722238 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.722189    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc"]
Jul 27 17:39:46.722412 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.722303    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc"
Jul 27 17:39:46.724981 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.724957    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 17:39:46.726098 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.726080    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-tlbjr\""
Jul 27 17:39:46.726179 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.726086    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:39:46.750380 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.750357    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpm42\" (UniqueName: \"kubernetes.io/projected/8a752062-406c-4a2d-9768-eff3ce0c2454-kube-api-access-qpm42\") pod \"openshift-lws-operator-fd8ccff4c-zv4dc\" (UID: \"8a752062-406c-4a2d-9768-eff3ce0c2454\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc"
Jul 27 17:39:46.750476 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.750395    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a752062-406c-4a2d-9768-eff3ce0c2454-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zv4dc\" (UID: \"8a752062-406c-4a2d-9768-eff3ce0c2454\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc"
Jul 27 17:39:46.851659 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.851628    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpm42\" (UniqueName: \"kubernetes.io/projected/8a752062-406c-4a2d-9768-eff3ce0c2454-kube-api-access-qpm42\") pod \"openshift-lws-operator-fd8ccff4c-zv4dc\" (UID: \"8a752062-406c-4a2d-9768-eff3ce0c2454\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc"
Jul 27 17:39:46.851826 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.851676    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a752062-406c-4a2d-9768-eff3ce0c2454-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zv4dc\" (UID: \"8a752062-406c-4a2d-9768-eff3ce0c2454\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc"
Jul 27 17:39:46.852016 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.851999    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a752062-406c-4a2d-9768-eff3ce0c2454-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zv4dc\" (UID: \"8a752062-406c-4a2d-9768-eff3ce0c2454\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc"
Jul 27 17:39:46.860295 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:46.860276    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpm42\" (UniqueName: \"kubernetes.io/projected/8a752062-406c-4a2d-9768-eff3ce0c2454-kube-api-access-qpm42\") pod \"openshift-lws-operator-fd8ccff4c-zv4dc\" (UID: \"8a752062-406c-4a2d-9768-eff3ce0c2454\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc"
Jul 27 17:39:47.031725 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:47.031688    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc"
Jul 27 17:39:47.171693 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:47.171669    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc"]
Jul 27 17:39:47.173362 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:39:47.173337    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a752062_406c_4a2d_9768_eff3ce0c2454.slice/crio-7d831efeb86d5b7cf5aa65c8e0c3995a2f6b2629a2d6428edc320f4e755ccb7a WatchSource:0}: Error finding container 7d831efeb86d5b7cf5aa65c8e0c3995a2f6b2629a2d6428edc320f4e755ccb7a: Status 404 returned error can't find the container with id 7d831efeb86d5b7cf5aa65c8e0c3995a2f6b2629a2d6428edc320f4e755ccb7a
Jul 27 17:39:47.736415 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:47.736379    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc" event={"ID":"8a752062-406c-4a2d-9768-eff3ce0c2454","Type":"ContainerStarted","Data":"7d831efeb86d5b7cf5aa65c8e0c3995a2f6b2629a2d6428edc320f4e755ccb7a"}
Jul 27 17:39:50.750620 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:50.750576    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc" event={"ID":"8a752062-406c-4a2d-9768-eff3ce0c2454","Type":"ContainerStarted","Data":"730f93610e1f1d1f6680d26eca08969b73f21e2ea16ba93b2554a14ac8101a04"}
Jul 27 17:39:50.770561 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:39:50.770509    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zv4dc" podStartSLOduration=1.601554803 podStartE2EDuration="4.770482339s" totalImagesPullingTime="3.168927536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:39:50.768314482 +0000 UTC m=+672.813052563" watchObservedRunningTime="2026-07-27 17:39:50.770482339 +0000 UTC m=+672.815220419"
Jul 27 17:40:05.969802 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:05.969768    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"]
Jul 27 17:40:05.977691 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:05.977658    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:05.980245 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:05.980217    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 17:40:05.981099 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:05.981067    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 17:40:05.981213 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:05.981133    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 17:40:05.981213 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:05.981133    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-l2fmb\""
Jul 27 17:40:05.987156 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:05.987133    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"]
Jul 27 17:40:06.100268 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.100231    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.100454 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.100281    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.100454 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.100303    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx4xq\" (UniqueName: \"kubernetes.io/projected/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-kube-api-access-gx4xq\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.100454 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.100321    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.100454 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.100357    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.100454 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.100372    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.100454 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.100431    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.100454 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.100449    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.100730 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.100465    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.201793 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.201751    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.201793 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.201797    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.202048 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.201848    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.202048 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.201879    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.202048 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.201901    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.202048 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.202029    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.202247 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.202084    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.202247 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.202112    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx4xq\" (UniqueName: \"kubernetes.io/projected/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-kube-api-access-gx4xq\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.202247 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.202132    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.202575 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.202552    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.202741 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.202580    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.202741 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.202617    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.202741 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.202632    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.202912 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.202642    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.204314 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.204291    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.204473 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.204458    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.210023 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.209997    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.210277 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.210257    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx4xq\" (UniqueName: \"kubernetes.io/projected/e27bde2a-4e4d-4ea9-9b27-4665b50a751c-kube-api-access-gx4xq\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-spfsb\" (UID: \"e27bde2a-4e4d-4ea9-9b27-4665b50a751c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.290223 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.290119    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:06.422784 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.422758    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"]
Jul 27 17:40:06.424799 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:40:06.424773    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27bde2a_4e4d_4ea9_9b27_4665b50a751c.slice/crio-401061541dd1d3fa15b4849684265a58e2022562ac631c0c2aa3fc78e973c812 WatchSource:0}: Error finding container 401061541dd1d3fa15b4849684265a58e2022562ac631c0c2aa3fc78e973c812: Status 404 returned error can't find the container with id 401061541dd1d3fa15b4849684265a58e2022562ac631c0c2aa3fc78e973c812
Jul 27 17:40:06.427322 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.427306    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:40:06.819102 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:06.819068    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb" event={"ID":"e27bde2a-4e4d-4ea9-9b27-4665b50a751c","Type":"ContainerStarted","Data":"401061541dd1d3fa15b4849684265a58e2022562ac631c0c2aa3fc78e973c812"}
Jul 27 17:40:08.935966 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:08.935906    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:40:08.936216 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:08.936014    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:40:08.936216 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:08.936046    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:40:09.837862 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:09.837825    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb" event={"ID":"e27bde2a-4e4d-4ea9-9b27-4665b50a751c","Type":"ContainerStarted","Data":"03dc198d63c94eec0aeef5045981412c47bc18d175f8504d984f85799d4c3f2c"}
Jul 27 17:40:09.858169 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:09.858116    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb" podStartSLOduration=2.349901903 podStartE2EDuration="4.858100145s" totalImagesPullingTime="2.508198242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:40:09.856960056 +0000 UTC m=+691.901698176" watchObservedRunningTime="2026-07-27 17:40:09.858100145 +0000 UTC m=+691.902838226"
Jul 27 17:40:10.290510 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:10.290467    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:10.295259 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:10.295232    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:10.841931 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:10.841897    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:10.842854 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:10.842834    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:40:14.889399 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:14.889364    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"]
Jul 27 17:40:14.892853 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:14.892836    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"
Jul 27 17:40:14.896639 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:14.896616    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 17:40:14.896775 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:14.896615    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 17:40:14.897849 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:14.897828    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xhmts\""
Jul 27 17:40:14.907193 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:14.907171    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"]
Jul 27 17:40:14.976205 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:14.976170    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj\" (UID: \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"
Jul 27 17:40:14.976205 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:14.976218    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wj6p\" (UniqueName: \"kubernetes.io/projected/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-kube-api-access-9wj6p\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj\" (UID: \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"
Jul 27 17:40:14.976463 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:14.976341    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj\" (UID: \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"
Jul 27 17:40:14.997737 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:14.997707    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"]
Jul 27 17:40:15.001000 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.000984    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"
Jul 27 17:40:15.009899 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.009875    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"]
Jul 27 17:40:15.077414 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.077380    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdbmx\" (UniqueName: \"kubernetes.io/projected/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-kube-api-access-tdbmx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf\" (UID: \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"
Jul 27 17:40:15.077642 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.077444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj\" (UID: \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"
Jul 27 17:40:15.077642 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.077540    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj\" (UID: \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"
Jul 27 17:40:15.077642 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.077579    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf\" (UID: \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"
Jul 27 17:40:15.077642 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.077607    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf\" (UID: \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"
Jul 27 17:40:15.077810 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.077644    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wj6p\" (UniqueName: \"kubernetes.io/projected/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-kube-api-access-9wj6p\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj\" (UID: \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"
Jul 27 17:40:15.077844 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.077811    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj\" (UID: \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"
Jul 27 17:40:15.077844 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.077831    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj\" (UID: \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"
Jul 27 17:40:15.089009 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.088978    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wj6p\" (UniqueName: \"kubernetes.io/projected/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-kube-api-access-9wj6p\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj\" (UID: \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"
Jul 27 17:40:15.095269 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.095246    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"]
Jul 27 17:40:15.099010 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.098995    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"
Jul 27 17:40:15.109342 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.109311    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"]
Jul 27 17:40:15.178587 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.178475    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7\" (UID: \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"
Jul 27 17:40:15.178587 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.178562    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdbmx\" (UniqueName: \"kubernetes.io/projected/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-kube-api-access-tdbmx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf\" (UID: \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"
Jul 27 17:40:15.178770 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.178599    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rkwm\" (UniqueName: \"kubernetes.io/projected/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-kube-api-access-5rkwm\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7\" (UID: \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"
Jul 27 17:40:15.178770 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.178645    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7\" (UID: \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"
Jul 27 17:40:15.178770 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.178668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf\" (UID: \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"
Jul 27 17:40:15.178770 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.178687    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf\" (UID: \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"
Jul 27 17:40:15.179169 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.179145    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf\" (UID: \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"
Jul 27 17:40:15.179205 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.179155    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf\" (UID: \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"
Jul 27 17:40:15.185884 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.185857    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"]
Jul 27 17:40:15.189242 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.189222    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"
Jul 27 17:40:15.191340 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.191316    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdbmx\" (UniqueName: \"kubernetes.io/projected/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-kube-api-access-tdbmx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf\" (UID: \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"
Jul 27 17:40:15.200894 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.200870    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"]
Jul 27 17:40:15.202002 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.201979    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"
Jul 27 17:40:15.279464 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.279427    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rkwm\" (UniqueName: \"kubernetes.io/projected/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-kube-api-access-5rkwm\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7\" (UID: \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"
Jul 27 17:40:15.279676 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.279486    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjqh\" (UniqueName: \"kubernetes.io/projected/9e0f4481-223b-490c-bf1d-18772b11ddf4-kube-api-access-ptjqh\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6\" (UID: \"9e0f4481-223b-490c-bf1d-18772b11ddf4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"
Jul 27 17:40:15.279676 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.279552    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e0f4481-223b-490c-bf1d-18772b11ddf4-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6\" (UID: \"9e0f4481-223b-490c-bf1d-18772b11ddf4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"
Jul 27 17:40:15.279676 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.279594    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7\" (UID: \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"
Jul 27 17:40:15.279676 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.279656    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e0f4481-223b-490c-bf1d-18772b11ddf4-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6\" (UID: \"9e0f4481-223b-490c-bf1d-18772b11ddf4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"
Jul 27 17:40:15.279884 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.279704    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7\" (UID: \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"
Jul 27 17:40:15.279997 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.279978    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7\" (UID: \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"
Jul 27 17:40:15.280030 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.279996    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7\" (UID: \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"
Jul 27 17:40:15.289295 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.289256    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rkwm\" (UniqueName: \"kubernetes.io/projected/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-kube-api-access-5rkwm\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7\" (UID: \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"
Jul 27 17:40:15.310060 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.310031    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"
Jul 27 17:40:15.334192 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.334109    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"]
Jul 27 17:40:15.336401 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:40:15.336379    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca6e0ac_c0ae_4447_a24d_c7c335a17bdd.slice/crio-8165c606e4a93244e5b903cffc5b019ae3117644bd349240d4b10d8ebc9033e0 WatchSource:0}: Error finding container 8165c606e4a93244e5b903cffc5b019ae3117644bd349240d4b10d8ebc9033e0: Status 404 returned error can't find the container with id 8165c606e4a93244e5b903cffc5b019ae3117644bd349240d4b10d8ebc9033e0
Jul 27 17:40:15.380113 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.380076    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjqh\" (UniqueName: \"kubernetes.io/projected/9e0f4481-223b-490c-bf1d-18772b11ddf4-kube-api-access-ptjqh\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6\" (UID: \"9e0f4481-223b-490c-bf1d-18772b11ddf4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"
Jul 27 17:40:15.380275 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.380124    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e0f4481-223b-490c-bf1d-18772b11ddf4-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6\" (UID: \"9e0f4481-223b-490c-bf1d-18772b11ddf4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"
Jul 27 17:40:15.380275 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.380172    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e0f4481-223b-490c-bf1d-18772b11ddf4-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6\" (UID: \"9e0f4481-223b-490c-bf1d-18772b11ddf4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"
Jul 27 17:40:15.380540 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.380518    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e0f4481-223b-490c-bf1d-18772b11ddf4-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6\" (UID: \"9e0f4481-223b-490c-bf1d-18772b11ddf4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"
Jul 27 17:40:15.380648 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.380545    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e0f4481-223b-490c-bf1d-18772b11ddf4-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6\" (UID: \"9e0f4481-223b-490c-bf1d-18772b11ddf4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"
Jul 27 17:40:15.389715 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.389693    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjqh\" (UniqueName: \"kubernetes.io/projected/9e0f4481-223b-490c-bf1d-18772b11ddf4-kube-api-access-ptjqh\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6\" (UID: \"9e0f4481-223b-490c-bf1d-18772b11ddf4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"
Jul 27 17:40:15.414471 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.414252    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"
Jul 27 17:40:15.448752 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.448728    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"]
Jul 27 17:40:15.451370 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:40:15.451335    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8723cd0a_53a8_4f0f_ae02_3a8afd97564d.slice/crio-7d711fbd57a6c755576485b1bb90e8f4ae5d94960fc1aca4800e89a02f97bdd4 WatchSource:0}: Error finding container 7d711fbd57a6c755576485b1bb90e8f4ae5d94960fc1aca4800e89a02f97bdd4: Status 404 returned error can't find the container with id 7d711fbd57a6c755576485b1bb90e8f4ae5d94960fc1aca4800e89a02f97bdd4
Jul 27 17:40:15.505463 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.505433    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"
Jul 27 17:40:15.550883 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.550853    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"]
Jul 27 17:40:15.553874 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:40:15.553842    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ad39ef_3dfe_4d3f_ae19_a51ef6a0b6a1.slice/crio-bc5f0e4b86b901dd32dd7164de37c87ab681a2b509e74e1a6c89aa33b558f7f3 WatchSource:0}: Error finding container bc5f0e4b86b901dd32dd7164de37c87ab681a2b509e74e1a6c89aa33b558f7f3: Status 404 returned error can't find the container with id bc5f0e4b86b901dd32dd7164de37c87ab681a2b509e74e1a6c89aa33b558f7f3
Jul 27 17:40:15.644820 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.644795    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"]
Jul 27 17:40:15.647996 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:40:15.647970    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0f4481_223b_490c_bf1d_18772b11ddf4.slice/crio-73f064ad07b9b9563e1764e912d08d85673c7bf5ebaf7794cc679bf2ef58abf3 WatchSource:0}: Error finding container 73f064ad07b9b9563e1764e912d08d85673c7bf5ebaf7794cc679bf2ef58abf3: Status 404 returned error can't find the container with id 73f064ad07b9b9563e1764e912d08d85673c7bf5ebaf7794cc679bf2ef58abf3
Jul 27 17:40:15.863659 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.863618    2571 generic.go:356] "Generic (PLEG): container finished" podID="cca6e0ac-c0ae-4447-a24d-c7c335a17bdd" containerID="54bbbc2621d748e5b5f605ef89fa99ed31c6e5bc6e0a3aeed9382b59010737cf" exitCode=0
Jul 27 17:40:15.863831 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.863702    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj" event={"ID":"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd","Type":"ContainerDied","Data":"54bbbc2621d748e5b5f605ef89fa99ed31c6e5bc6e0a3aeed9382b59010737cf"}
Jul 27 17:40:15.863831 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.863759    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj" event={"ID":"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd","Type":"ContainerStarted","Data":"8165c606e4a93244e5b903cffc5b019ae3117644bd349240d4b10d8ebc9033e0"}
Jul 27 17:40:15.865275 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.865256    2571 generic.go:356] "Generic (PLEG): container finished" podID="8723cd0a-53a8-4f0f-ae02-3a8afd97564d" containerID="a14afa8abae5e920777b866e263debdedf898dfc05eb8d7326fe58b69d239a91" exitCode=0
Jul 27 17:40:15.865378 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.865345    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf" event={"ID":"8723cd0a-53a8-4f0f-ae02-3a8afd97564d","Type":"ContainerDied","Data":"a14afa8abae5e920777b866e263debdedf898dfc05eb8d7326fe58b69d239a91"}
Jul 27 17:40:15.865518 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.865400    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf" event={"ID":"8723cd0a-53a8-4f0f-ae02-3a8afd97564d","Type":"ContainerStarted","Data":"7d711fbd57a6c755576485b1bb90e8f4ae5d94960fc1aca4800e89a02f97bdd4"}
Jul 27 17:40:15.866897 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.866871    2571 generic.go:356] "Generic (PLEG): container finished" podID="07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1" containerID="f6aa29f73d374ce59248c171d08393d8a6effb5c09caa414bf49910f03b6fd54" exitCode=0
Jul 27 17:40:15.866996 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.866910    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7" event={"ID":"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1","Type":"ContainerDied","Data":"f6aa29f73d374ce59248c171d08393d8a6effb5c09caa414bf49910f03b6fd54"}
Jul 27 17:40:15.866996 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.866947    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7" event={"ID":"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1","Type":"ContainerStarted","Data":"bc5f0e4b86b901dd32dd7164de37c87ab681a2b509e74e1a6c89aa33b558f7f3"}
Jul 27 17:40:15.868745 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.868726    2571 generic.go:356] "Generic (PLEG): container finished" podID="9e0f4481-223b-490c-bf1d-18772b11ddf4" containerID="d33cf16f8f19578997d3359a09ed9db8a2d9dcb4dab414d568c2d763bfa6f473" exitCode=0
Jul 27 17:40:15.868831 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.868771    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6" event={"ID":"9e0f4481-223b-490c-bf1d-18772b11ddf4","Type":"ContainerDied","Data":"d33cf16f8f19578997d3359a09ed9db8a2d9dcb4dab414d568c2d763bfa6f473"}
Jul 27 17:40:15.868831 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:15.868809    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6" event={"ID":"9e0f4481-223b-490c-bf1d-18772b11ddf4","Type":"ContainerStarted","Data":"73f064ad07b9b9563e1764e912d08d85673c7bf5ebaf7794cc679bf2ef58abf3"}
Jul 27 17:40:17.102009 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.101978    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"]
Jul 27 17:40:17.105272 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.105252    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:17.118837 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:40:17.118808    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"metrics-server-cert\" is forbidden: User \"system:node:ip-10-0-131-74.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-131-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" type="*v1.Secret"
Jul 27 17:40:17.118925 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:40:17.118904    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"lws-controller-manager-67d9b7fdbd-hckgz\" is forbidden: User \"system:node:ip-10-0-131-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-131-74.ec2.internal' and this object" podUID="d32fca4b-75e0-4b3c-a25d-b9352215d749" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:17.119143 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:40:17.119094    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"lws-controller-manager-dockercfg-hvdmn\" is forbidden: User \"system:node:ip-10-0-131-74.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-131-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hvdmn\"" type="*v1.Secret"
Jul 27 17:40:17.119286 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:40:17.119262    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"webhook-server-cert\" is forbidden: User \"system:node:ip-10-0-131-74.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-131-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" type="*v1.Secret"
Jul 27 17:40:17.119286 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:40:17.119263    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"lws-manager-config\" is forbidden: User \"system:node:ip-10-0-131-74.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-131-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" type="*v1.ConfigMap"
Jul 27 17:40:17.124233 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.124212    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"]
Jul 27 17:40:17.194893 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.194863    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d32fca4b-75e0-4b3c-a25d-b9352215d749-metrics-cert\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:17.195001 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.194922    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d32fca4b-75e0-4b3c-a25d-b9352215d749-manager-config\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:17.195001 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.194976    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pcsg\" (UniqueName: \"kubernetes.io/projected/d32fca4b-75e0-4b3c-a25d-b9352215d749-kube-api-access-2pcsg\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:17.195069 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.195017    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d32fca4b-75e0-4b3c-a25d-b9352215d749-cert\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:17.295966 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.295918    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d32fca4b-75e0-4b3c-a25d-b9352215d749-manager-config\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:17.295966 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.295970    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pcsg\" (UniqueName: \"kubernetes.io/projected/d32fca4b-75e0-4b3c-a25d-b9352215d749-kube-api-access-2pcsg\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:17.296305 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.296074    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d32fca4b-75e0-4b3c-a25d-b9352215d749-cert\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:17.296305 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.296162    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d32fca4b-75e0-4b3c-a25d-b9352215d749-metrics-cert\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:17.319702 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.319670    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pcsg\" (UniqueName: \"kubernetes.io/projected/d32fca4b-75e0-4b3c-a25d-b9352215d749-kube-api-access-2pcsg\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:17.879307 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.879271    2571 generic.go:356] "Generic (PLEG): container finished" podID="9e0f4481-223b-490c-bf1d-18772b11ddf4" containerID="4fe1c7fd8c4cca35fa60da86bf377539fa5acb04fd5ef389bff2c906ce7ec9d0" exitCode=0
Jul 27 17:40:17.879622 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.879357    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6" event={"ID":"9e0f4481-223b-490c-bf1d-18772b11ddf4","Type":"ContainerDied","Data":"4fe1c7fd8c4cca35fa60da86bf377539fa5acb04fd5ef389bff2c906ce7ec9d0"}
Jul 27 17:40:17.881209 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.881188    2571 generic.go:356] "Generic (PLEG): container finished" podID="cca6e0ac-c0ae-4447-a24d-c7c335a17bdd" containerID="5af020cee2337b03a0b30ad01dd85422b96ad07bb9cf716848577af5f1ddbe52" exitCode=0
Jul 27 17:40:17.881296 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:17.881261    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj" event={"ID":"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd","Type":"ContainerDied","Data":"5af020cee2337b03a0b30ad01dd85422b96ad07bb9cf716848577af5f1ddbe52"}
Jul 27 17:40:18.296975 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:40:18.296944    2571 secret.go:189] Couldn't get secret openshift-lws-operator/metrics-server-cert: failed to sync secret cache: timed out waiting for the condition
Jul 27 17:40:18.296975 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:40:18.296972    2571 secret.go:189] Couldn't get secret openshift-lws-operator/webhook-server-cert: failed to sync secret cache: timed out waiting for the condition
Jul 27 17:40:18.297297 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:40:18.296954    2571 configmap.go:193] Couldn't get configMap openshift-lws-operator/lws-manager-config: failed to sync configmap cache: timed out waiting for the condition
Jul 27 17:40:18.297297 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:40:18.297034    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32fca4b-75e0-4b3c-a25d-b9352215d749-cert podName:d32fca4b-75e0-4b3c-a25d-b9352215d749 nodeName:}" failed. No retries permitted until 2026-07-27 17:40:18.797013029 +0000 UTC m=+700.841751087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d32fca4b-75e0-4b3c-a25d-b9352215d749-cert") pod "lws-controller-manager-67d9b7fdbd-hckgz" (UID: "d32fca4b-75e0-4b3c-a25d-b9352215d749") : failed to sync secret cache: timed out waiting for the condition
Jul 27 17:40:18.297297 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:40:18.297051    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d32fca4b-75e0-4b3c-a25d-b9352215d749-manager-config podName:d32fca4b-75e0-4b3c-a25d-b9352215d749 nodeName:}" failed. No retries permitted until 2026-07-27 17:40:18.797042735 +0000 UTC m=+700.841780793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "manager-config" (UniqueName: "kubernetes.io/configmap/d32fca4b-75e0-4b3c-a25d-b9352215d749-manager-config") pod "lws-controller-manager-67d9b7fdbd-hckgz" (UID: "d32fca4b-75e0-4b3c-a25d-b9352215d749") : failed to sync configmap cache: timed out waiting for the condition
Jul 27 17:40:18.297297 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:40:18.297064    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32fca4b-75e0-4b3c-a25d-b9352215d749-metrics-cert podName:d32fca4b-75e0-4b3c-a25d-b9352215d749 nodeName:}" failed. No retries permitted until 2026-07-27 17:40:18.797057396 +0000 UTC m=+700.841795455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-cert" (UniqueName: "kubernetes.io/secret/d32fca4b-75e0-4b3c-a25d-b9352215d749-metrics-cert") pod "lws-controller-manager-67d9b7fdbd-hckgz" (UID: "d32fca4b-75e0-4b3c-a25d-b9352215d749") : failed to sync secret cache: timed out waiting for the condition
Jul 27 17:40:18.308335 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.308308    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hvdmn\""
Jul 27 17:40:18.318443 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.318423    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 17:40:18.542579 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.542551    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 17:40:18.671266 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.671238    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 17:40:18.812355 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.812269    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d32fca4b-75e0-4b3c-a25d-b9352215d749-metrics-cert\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:18.812355 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.812334    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d32fca4b-75e0-4b3c-a25d-b9352215d749-manager-config\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:18.812637 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.812361    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d32fca4b-75e0-4b3c-a25d-b9352215d749-cert\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:18.813156 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.813124    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d32fca4b-75e0-4b3c-a25d-b9352215d749-manager-config\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:18.814795 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.814776    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d32fca4b-75e0-4b3c-a25d-b9352215d749-metrics-cert\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:18.814857 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.814792    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d32fca4b-75e0-4b3c-a25d-b9352215d749-cert\") pod \"lws-controller-manager-67d9b7fdbd-hckgz\" (UID: \"d32fca4b-75e0-4b3c-a25d-b9352215d749\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:18.887881 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.887846    2571 generic.go:356] "Generic (PLEG): container finished" podID="cca6e0ac-c0ae-4447-a24d-c7c335a17bdd" containerID="307b59a6002ea5d752b461d3e7007f85643895d1466d0a19483d4f8aad5fae28" exitCode=0
Jul 27 17:40:18.888053 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.887911    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj" event={"ID":"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd","Type":"ContainerDied","Data":"307b59a6002ea5d752b461d3e7007f85643895d1466d0a19483d4f8aad5fae28"}
Jul 27 17:40:18.889806 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.889780    2571 generic.go:356] "Generic (PLEG): container finished" podID="8723cd0a-53a8-4f0f-ae02-3a8afd97564d" containerID="2a2332c9707a39abc064dcaf5d526c43861ca633989233577357d164d97aa2bd" exitCode=0
Jul 27 17:40:18.889929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.889864    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf" event={"ID":"8723cd0a-53a8-4f0f-ae02-3a8afd97564d","Type":"ContainerDied","Data":"2a2332c9707a39abc064dcaf5d526c43861ca633989233577357d164d97aa2bd"}
Jul 27 17:40:18.891821 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.891802    2571 generic.go:356] "Generic (PLEG): container finished" podID="07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1" containerID="e9cb1d0ccaa245e7644f77291eb187a6d0e2e124a915cf467bffa1d5e9988b78" exitCode=0
Jul 27 17:40:18.891925 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.891886    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7" event={"ID":"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1","Type":"ContainerDied","Data":"e9cb1d0ccaa245e7644f77291eb187a6d0e2e124a915cf467bffa1d5e9988b78"}
Jul 27 17:40:18.894066 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.894048    2571 generic.go:356] "Generic (PLEG): container finished" podID="9e0f4481-223b-490c-bf1d-18772b11ddf4" containerID="36139f518098cc85bdced51de51495a07d41a85e2a1edb05066bead577ab835b" exitCode=0
Jul 27 17:40:18.894172 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.894092    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6" event={"ID":"9e0f4481-223b-490c-bf1d-18772b11ddf4","Type":"ContainerDied","Data":"36139f518098cc85bdced51de51495a07d41a85e2a1edb05066bead577ab835b"}
Jul 27 17:40:18.915373 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:18.915352    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:19.069147 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:19.069121    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"]
Jul 27 17:40:19.108432 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:40:19.108401    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32fca4b_75e0_4b3c_a25d_b9352215d749.slice/crio-80505be940a4599b272b0fe380adf2e2e09cd16560237662a38de782a7dbc5d2 WatchSource:0}: Error finding container 80505be940a4599b272b0fe380adf2e2e09cd16560237662a38de782a7dbc5d2: Status 404 returned error can't find the container with id 80505be940a4599b272b0fe380adf2e2e09cd16560237662a38de782a7dbc5d2
Jul 27 17:40:19.901156 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:19.901120    2571 generic.go:356] "Generic (PLEG): container finished" podID="8723cd0a-53a8-4f0f-ae02-3a8afd97564d" containerID="7e042ee35df69f6fcb4fa42cd000b65fa2bf060a190f5fa3b98b699f0ea0371d" exitCode=0
Jul 27 17:40:19.901631 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:19.901205    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf" event={"ID":"8723cd0a-53a8-4f0f-ae02-3a8afd97564d","Type":"ContainerDied","Data":"7e042ee35df69f6fcb4fa42cd000b65fa2bf060a190f5fa3b98b699f0ea0371d"}
Jul 27 17:40:19.903220 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:19.903194    2571 generic.go:356] "Generic (PLEG): container finished" podID="07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1" containerID="1b2ed4f1197512bfd1706fd63cf4cc3556ef31d35441cef456c825d19906bb94" exitCode=0
Jul 27 17:40:19.903341 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:19.903280    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7" event={"ID":"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1","Type":"ContainerDied","Data":"1b2ed4f1197512bfd1706fd63cf4cc3556ef31d35441cef456c825d19906bb94"}
Jul 27 17:40:19.904440 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:19.904413    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz" event={"ID":"d32fca4b-75e0-4b3c-a25d-b9352215d749","Type":"ContainerStarted","Data":"80505be940a4599b272b0fe380adf2e2e09cd16560237662a38de782a7dbc5d2"}
Jul 27 17:40:20.034409 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.034383    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"
Jul 27 17:40:20.064114 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.064091    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"
Jul 27 17:40:20.125160 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.125131    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-util\") pod \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\" (UID: \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\") "
Jul 27 17:40:20.125331 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.125248    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-bundle\") pod \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\" (UID: \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\") "
Jul 27 17:40:20.125331 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.125304    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wj6p\" (UniqueName: \"kubernetes.io/projected/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-kube-api-access-9wj6p\") pod \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\" (UID: \"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd\") "
Jul 27 17:40:20.125750 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.125726    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-bundle" (OuterVolumeSpecName: "bundle") pod "cca6e0ac-c0ae-4447-a24d-c7c335a17bdd" (UID: "cca6e0ac-c0ae-4447-a24d-c7c335a17bdd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:40:20.127364 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.127345    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-kube-api-access-9wj6p" (OuterVolumeSpecName: "kube-api-access-9wj6p") pod "cca6e0ac-c0ae-4447-a24d-c7c335a17bdd" (UID: "cca6e0ac-c0ae-4447-a24d-c7c335a17bdd"). InnerVolumeSpecName "kube-api-access-9wj6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:40:20.130720 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.130696    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-util" (OuterVolumeSpecName: "util") pod "cca6e0ac-c0ae-4447-a24d-c7c335a17bdd" (UID: "cca6e0ac-c0ae-4447-a24d-c7c335a17bdd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:40:20.226095 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.226058    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e0f4481-223b-490c-bf1d-18772b11ddf4-bundle\") pod \"9e0f4481-223b-490c-bf1d-18772b11ddf4\" (UID: \"9e0f4481-223b-490c-bf1d-18772b11ddf4\") "
Jul 27 17:40:20.226095 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.226095    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e0f4481-223b-490c-bf1d-18772b11ddf4-util\") pod \"9e0f4481-223b-490c-bf1d-18772b11ddf4\" (UID: \"9e0f4481-223b-490c-bf1d-18772b11ddf4\") "
Jul 27 17:40:20.226309 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.226131    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptjqh\" (UniqueName: \"kubernetes.io/projected/9e0f4481-223b-490c-bf1d-18772b11ddf4-kube-api-access-ptjqh\") pod \"9e0f4481-223b-490c-bf1d-18772b11ddf4\" (UID: \"9e0f4481-223b-490c-bf1d-18772b11ddf4\") "
Jul 27 17:40:20.226375 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.226341    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-bundle\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:40:20.226375 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.226358    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wj6p\" (UniqueName: \"kubernetes.io/projected/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-kube-api-access-9wj6p\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:40:20.226481 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.226374    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cca6e0ac-c0ae-4447-a24d-c7c335a17bdd-util\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:40:20.226678 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.226655    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0f4481-223b-490c-bf1d-18772b11ddf4-bundle" (OuterVolumeSpecName: "bundle") pod "9e0f4481-223b-490c-bf1d-18772b11ddf4" (UID: "9e0f4481-223b-490c-bf1d-18772b11ddf4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:40:20.228235 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.228215    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0f4481-223b-490c-bf1d-18772b11ddf4-kube-api-access-ptjqh" (OuterVolumeSpecName: "kube-api-access-ptjqh") pod "9e0f4481-223b-490c-bf1d-18772b11ddf4" (UID: "9e0f4481-223b-490c-bf1d-18772b11ddf4"). InnerVolumeSpecName "kube-api-access-ptjqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:40:20.233521 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.233483    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0f4481-223b-490c-bf1d-18772b11ddf4-util" (OuterVolumeSpecName: "util") pod "9e0f4481-223b-490c-bf1d-18772b11ddf4" (UID: "9e0f4481-223b-490c-bf1d-18772b11ddf4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:40:20.326980 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.326947    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e0f4481-223b-490c-bf1d-18772b11ddf4-bundle\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:40:20.326980 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.326973    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e0f4481-223b-490c-bf1d-18772b11ddf4-util\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:40:20.326980 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.326982    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptjqh\" (UniqueName: \"kubernetes.io/projected/9e0f4481-223b-490c-bf1d-18772b11ddf4-kube-api-access-ptjqh\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:40:20.912393 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.912359    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj"
Jul 27 17:40:20.912393 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.912382    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183clnqwj" event={"ID":"cca6e0ac-c0ae-4447-a24d-c7c335a17bdd","Type":"ContainerDied","Data":"8165c606e4a93244e5b903cffc5b019ae3117644bd349240d4b10d8ebc9033e0"}
Jul 27 17:40:20.912884 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.912431    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8165c606e4a93244e5b903cffc5b019ae3117644bd349240d4b10d8ebc9033e0"
Jul 27 17:40:20.914529 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.914481    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6" event={"ID":"9e0f4481-223b-490c-bf1d-18772b11ddf4","Type":"ContainerDied","Data":"73f064ad07b9b9563e1764e912d08d85673c7bf5ebaf7794cc679bf2ef58abf3"}
Jul 27 17:40:20.914687 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.914544    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73f064ad07b9b9563e1764e912d08d85673c7bf5ebaf7794cc679bf2ef58abf3"
Jul 27 17:40:20.914751 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:20.914694    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgcgd6"
Jul 27 17:40:21.047810 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.047785    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"
Jul 27 17:40:21.064603 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.064581    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"
Jul 27 17:40:21.134423 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.134391    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdbmx\" (UniqueName: \"kubernetes.io/projected/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-kube-api-access-tdbmx\") pod \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\" (UID: \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\") "
Jul 27 17:40:21.134598 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.134438    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rkwm\" (UniqueName: \"kubernetes.io/projected/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-kube-api-access-5rkwm\") pod \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\" (UID: \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\") "
Jul 27 17:40:21.134598 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.134472    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-bundle\") pod \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\" (UID: \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\") "
Jul 27 17:40:21.134598 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.134514    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-bundle\") pod \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\" (UID: \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\") "
Jul 27 17:40:21.134598 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.134569    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-util\") pod \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\" (UID: \"8723cd0a-53a8-4f0f-ae02-3a8afd97564d\") "
Jul 27 17:40:21.134810 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.134617    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-util\") pod \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\" (UID: \"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1\") "
Jul 27 17:40:21.135065 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.135029    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-bundle" (OuterVolumeSpecName: "bundle") pod "8723cd0a-53a8-4f0f-ae02-3a8afd97564d" (UID: "8723cd0a-53a8-4f0f-ae02-3a8afd97564d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:40:21.135371 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.135346    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-bundle" (OuterVolumeSpecName: "bundle") pod "07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1" (UID: "07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:40:21.136690 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.136670    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-kube-api-access-5rkwm" (OuterVolumeSpecName: "kube-api-access-5rkwm") pod "07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1" (UID: "07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1"). InnerVolumeSpecName "kube-api-access-5rkwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:40:21.136890 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.136869    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-kube-api-access-tdbmx" (OuterVolumeSpecName: "kube-api-access-tdbmx") pod "8723cd0a-53a8-4f0f-ae02-3a8afd97564d" (UID: "8723cd0a-53a8-4f0f-ae02-3a8afd97564d"). InnerVolumeSpecName "kube-api-access-tdbmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:40:21.139724 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.139705    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-util" (OuterVolumeSpecName: "util") pod "07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1" (UID: "07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:40:21.140248 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.140230    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-util" (OuterVolumeSpecName: "util") pod "8723cd0a-53a8-4f0f-ae02-3a8afd97564d" (UID: "8723cd0a-53a8-4f0f-ae02-3a8afd97564d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:40:21.235775 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.235743    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-util\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:40:21.235775 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.235777    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdbmx\" (UniqueName: \"kubernetes.io/projected/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-kube-api-access-tdbmx\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:40:21.235982 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.235792    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rkwm\" (UniqueName: \"kubernetes.io/projected/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-kube-api-access-5rkwm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:40:21.235982 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.235805    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-bundle\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:40:21.235982 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.235816    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1-bundle\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:40:21.235982 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.235824    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8723cd0a-53a8-4f0f-ae02-3a8afd97564d-util\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:40:21.922414 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.922386    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7"
Jul 27 17:40:21.922844 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.922416    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053cm5f7" event={"ID":"07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1","Type":"ContainerDied","Data":"bc5f0e4b86b901dd32dd7164de37c87ab681a2b509e74e1a6c89aa33b558f7f3"}
Jul 27 17:40:21.922844 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.922463    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc5f0e4b86b901dd32dd7164de37c87ab681a2b509e74e1a6c89aa33b558f7f3"
Jul 27 17:40:21.924175 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.924152    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf" event={"ID":"8723cd0a-53a8-4f0f-ae02-3a8afd97564d","Type":"ContainerDied","Data":"7d711fbd57a6c755576485b1bb90e8f4ae5d94960fc1aca4800e89a02f97bdd4"}
Jul 27 17:40:21.924309 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.924186    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d711fbd57a6c755576485b1bb90e8f4ae5d94960fc1aca4800e89a02f97bdd4"
Jul 27 17:40:21.924309 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:21.924195    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a08mmdf"
Jul 27 17:40:22.929815 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:22.929773    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz" event={"ID":"d32fca4b-75e0-4b3c-a25d-b9352215d749","Type":"ContainerStarted","Data":"e3f37cc402268c9fb633b92eaeb2915d1d61fb5a33c3437f348b35a5e94c782a"}
Jul 27 17:40:22.930226 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:22.929837    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:33.936798 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:33.936721    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz"
Jul 27 17:40:33.961021 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:40:33.960972    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-hckgz" podStartSLOduration=14.126869285 podStartE2EDuration="16.96095767s" totalImagesPullingTime="2.834088385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:40:22.955297232 +0000 UTC m=+705.000035312" watchObservedRunningTime="2026-07-27 17:40:33.96095767 +0000 UTC m=+716.005695750"
Jul 27 17:41:19.731016 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.730982    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-xr2fb"]
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731276    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e0f4481-223b-490c-bf1d-18772b11ddf4" containerName="pull"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731287    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e0f4481-223b-490c-bf1d-18772b11ddf4" containerName="pull"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731299    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca6e0ac-c0ae-4447-a24d-c7c335a17bdd" containerName="extract"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731305    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca6e0ac-c0ae-4447-a24d-c7c335a17bdd" containerName="extract"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731312    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1" containerName="extract"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731317    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1" containerName="extract"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731325    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca6e0ac-c0ae-4447-a24d-c7c335a17bdd" containerName="util"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731330    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca6e0ac-c0ae-4447-a24d-c7c335a17bdd" containerName="util"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731337    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8723cd0a-53a8-4f0f-ae02-3a8afd97564d" containerName="pull"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731342    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8723cd0a-53a8-4f0f-ae02-3a8afd97564d" containerName="pull"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731354    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8723cd0a-53a8-4f0f-ae02-3a8afd97564d" containerName="extract"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731359    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8723cd0a-53a8-4f0f-ae02-3a8afd97564d" containerName="extract"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731366    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8723cd0a-53a8-4f0f-ae02-3a8afd97564d" containerName="util"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731371    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8723cd0a-53a8-4f0f-ae02-3a8afd97564d" containerName="util"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731376    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1" containerName="util"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731381    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1" containerName="util"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731387    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e0f4481-223b-490c-bf1d-18772b11ddf4" containerName="util"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731392    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e0f4481-223b-490c-bf1d-18772b11ddf4" containerName="util"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731397    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca6e0ac-c0ae-4447-a24d-c7c335a17bdd" containerName="pull"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731402    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca6e0ac-c0ae-4447-a24d-c7c335a17bdd" containerName="pull"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731407    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e0f4481-223b-490c-bf1d-18772b11ddf4" containerName="extract"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731412    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e0f4481-223b-490c-bf1d-18772b11ddf4" containerName="extract"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731417    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1" containerName="pull"
Jul 27 17:41:19.731434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731422    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1" containerName="pull"
Jul 27 17:41:19.732315 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731469    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="07ad39ef-3dfe-4d3f-ae19-a51ef6a0b6a1" containerName="extract"
Jul 27 17:41:19.732315 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731475    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8723cd0a-53a8-4f0f-ae02-3a8afd97564d" containerName="extract"
Jul 27 17:41:19.732315 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731482    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e0f4481-223b-490c-bf1d-18772b11ddf4" containerName="extract"
Jul 27 17:41:19.732315 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.731488    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cca6e0ac-c0ae-4447-a24d-c7c335a17bdd" containerName="extract"
Jul 27 17:41:19.734473 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.734457    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-xr2fb"
Jul 27 17:41:19.737173 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.737147    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 17:41:19.737268 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.737196    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 17:41:19.738314 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.738298    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sszrd\""
Jul 27 17:41:19.746525 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.746486    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-xr2fb"]
Jul 27 17:41:19.863239 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.863208    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6zkw2"]
Jul 27 17:41:19.866339 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.866324    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-6zkw2"
Jul 27 17:41:19.877928 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.877907    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6zkw2"]
Jul 27 17:41:19.916832 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:19.916805    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rcwc\" (UniqueName: \"kubernetes.io/projected/7fe14455-6123-40c1-801b-5ff8888894a5-kube-api-access-6rcwc\") pod \"authorino-b54df94b4-xr2fb\" (UID: \"7fe14455-6123-40c1-801b-5ff8888894a5\") " pod="kuadrant-system/authorino-b54df94b4-xr2fb"
Jul 27 17:41:20.018313 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:20.018228    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cztl4\" (UniqueName: \"kubernetes.io/projected/9180da43-c5ef-4b79-b9d8-3a7b690d53b4-kube-api-access-cztl4\") pod \"authorino-c548b5d7d-6zkw2\" (UID: \"9180da43-c5ef-4b79-b9d8-3a7b690d53b4\") " pod="kuadrant-system/authorino-c548b5d7d-6zkw2"
Jul 27 17:41:20.018313 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:20.018271    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rcwc\" (UniqueName: \"kubernetes.io/projected/7fe14455-6123-40c1-801b-5ff8888894a5-kube-api-access-6rcwc\") pod \"authorino-b54df94b4-xr2fb\" (UID: \"7fe14455-6123-40c1-801b-5ff8888894a5\") " pod="kuadrant-system/authorino-b54df94b4-xr2fb"
Jul 27 17:41:20.027485 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:20.027455    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rcwc\" (UniqueName: \"kubernetes.io/projected/7fe14455-6123-40c1-801b-5ff8888894a5-kube-api-access-6rcwc\") pod \"authorino-b54df94b4-xr2fb\" (UID: \"7fe14455-6123-40c1-801b-5ff8888894a5\") " pod="kuadrant-system/authorino-b54df94b4-xr2fb"
Jul 27 17:41:20.043268 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:20.043248    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-xr2fb"
Jul 27 17:41:20.119423 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:20.119390    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztl4\" (UniqueName: \"kubernetes.io/projected/9180da43-c5ef-4b79-b9d8-3a7b690d53b4-kube-api-access-cztl4\") pod \"authorino-c548b5d7d-6zkw2\" (UID: \"9180da43-c5ef-4b79-b9d8-3a7b690d53b4\") " pod="kuadrant-system/authorino-c548b5d7d-6zkw2"
Jul 27 17:41:20.128047 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:20.128018    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cztl4\" (UniqueName: \"kubernetes.io/projected/9180da43-c5ef-4b79-b9d8-3a7b690d53b4-kube-api-access-cztl4\") pod \"authorino-c548b5d7d-6zkw2\" (UID: \"9180da43-c5ef-4b79-b9d8-3a7b690d53b4\") " pod="kuadrant-system/authorino-c548b5d7d-6zkw2"
Jul 27 17:41:20.174099 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:20.174038    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-xr2fb"]
Jul 27 17:41:20.174404 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:20.174386    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-6zkw2"
Jul 27 17:41:20.178227 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:41:20.178203    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fe14455_6123_40c1_801b_5ff8888894a5.slice/crio-4f34f58268b7992d6f7e7dec0362cc375e6da70c75404ad3ef2645ba57790df1 WatchSource:0}: Error finding container 4f34f58268b7992d6f7e7dec0362cc375e6da70c75404ad3ef2645ba57790df1: Status 404 returned error can't find the container with id 4f34f58268b7992d6f7e7dec0362cc375e6da70c75404ad3ef2645ba57790df1
Jul 27 17:41:20.190067 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:20.190040    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-xr2fb" event={"ID":"7fe14455-6123-40c1-801b-5ff8888894a5","Type":"ContainerStarted","Data":"4f34f58268b7992d6f7e7dec0362cc375e6da70c75404ad3ef2645ba57790df1"}
Jul 27 17:41:20.301933 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:20.301905    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6zkw2"]
Jul 27 17:41:20.303741 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:41:20.303716    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9180da43_c5ef_4b79_b9d8_3a7b690d53b4.slice/crio-deb263231a4a763ef2058fe4803feb24512a5b0a90b70eb94ac457ad04ebe8ee WatchSource:0}: Error finding container deb263231a4a763ef2058fe4803feb24512a5b0a90b70eb94ac457ad04ebe8ee: Status 404 returned error can't find the container with id deb263231a4a763ef2058fe4803feb24512a5b0a90b70eb94ac457ad04ebe8ee
Jul 27 17:41:21.198969 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:21.198935    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-6zkw2" event={"ID":"9180da43-c5ef-4b79-b9d8-3a7b690d53b4","Type":"ContainerStarted","Data":"deb263231a4a763ef2058fe4803feb24512a5b0a90b70eb94ac457ad04ebe8ee"}
Jul 27 17:41:23.212564 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:23.212527    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-xr2fb" event={"ID":"7fe14455-6123-40c1-801b-5ff8888894a5","Type":"ContainerStarted","Data":"eed0f6b09096e25f5f6f030d62e8c2ac81d29ae259d583059e6329b44c2f9c11"}
Jul 27 17:41:23.231131 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:23.230899    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-xr2fb" podStartSLOduration=1.324764204 podStartE2EDuration="4.230880556s" totalImagesPullingTime="2.906116352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:41:23.228464122 +0000 UTC m=+765.273202202" watchObservedRunningTime="2026-07-27 17:41:23.230880556 +0000 UTC m=+765.275618638"
Jul 27 17:41:24.219541 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:24.219478    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-6zkw2" event={"ID":"9180da43-c5ef-4b79-b9d8-3a7b690d53b4","Type":"ContainerStarted","Data":"e77fde90cf0df5663dcfd47bfdd5795a2d977c20303781475075ea6244789e04"}
Jul 27 17:41:24.235568 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:24.235519    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-6zkw2" podStartSLOduration=2.042284268 podStartE2EDuration="5.235505911s" totalImagesPullingTime="3.193221643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:41:24.233891477 +0000 UTC m=+766.278629561" watchObservedRunningTime="2026-07-27 17:41:24.235505911 +0000 UTC m=+766.280243982"
Jul 27 17:41:24.262990 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:24.262961    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-xr2fb"]
Jul 27 17:41:25.224209 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:25.224172    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-xr2fb" podUID="7fe14455-6123-40c1-801b-5ff8888894a5" containerName="authorino" containerID="cri-o://eed0f6b09096e25f5f6f030d62e8c2ac81d29ae259d583059e6329b44c2f9c11" gracePeriod=30
Jul 27 17:41:25.465753 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:25.465729    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-xr2fb"
Jul 27 17:41:25.559060 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:25.558987    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rcwc\" (UniqueName: \"kubernetes.io/projected/7fe14455-6123-40c1-801b-5ff8888894a5-kube-api-access-6rcwc\") pod \"7fe14455-6123-40c1-801b-5ff8888894a5\" (UID: \"7fe14455-6123-40c1-801b-5ff8888894a5\") "
Jul 27 17:41:25.561088 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:25.561067    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe14455-6123-40c1-801b-5ff8888894a5-kube-api-access-6rcwc" (OuterVolumeSpecName: "kube-api-access-6rcwc") pod "7fe14455-6123-40c1-801b-5ff8888894a5" (UID: "7fe14455-6123-40c1-801b-5ff8888894a5"). InnerVolumeSpecName "kube-api-access-6rcwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:41:25.660079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:25.660049    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rcwc\" (UniqueName: \"kubernetes.io/projected/7fe14455-6123-40c1-801b-5ff8888894a5-kube-api-access-6rcwc\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:41:26.229896 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:26.229864    2571 generic.go:356] "Generic (PLEG): container finished" podID="7fe14455-6123-40c1-801b-5ff8888894a5" containerID="eed0f6b09096e25f5f6f030d62e8c2ac81d29ae259d583059e6329b44c2f9c11" exitCode=0
Jul 27 17:41:26.230293 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:26.229910    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-xr2fb"
Jul 27 17:41:26.230293 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:26.229955    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-xr2fb" event={"ID":"7fe14455-6123-40c1-801b-5ff8888894a5","Type":"ContainerDied","Data":"eed0f6b09096e25f5f6f030d62e8c2ac81d29ae259d583059e6329b44c2f9c11"}
Jul 27 17:41:26.230293 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:26.229995    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-xr2fb" event={"ID":"7fe14455-6123-40c1-801b-5ff8888894a5","Type":"ContainerDied","Data":"4f34f58268b7992d6f7e7dec0362cc375e6da70c75404ad3ef2645ba57790df1"}
Jul 27 17:41:26.230293 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:26.230014    2571 scope.go:117] "RemoveContainer" containerID="eed0f6b09096e25f5f6f030d62e8c2ac81d29ae259d583059e6329b44c2f9c11"
Jul 27 17:41:26.239138 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:26.239117    2571 scope.go:117] "RemoveContainer" containerID="eed0f6b09096e25f5f6f030d62e8c2ac81d29ae259d583059e6329b44c2f9c11"
Jul 27 17:41:26.239404 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:41:26.239380    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed0f6b09096e25f5f6f030d62e8c2ac81d29ae259d583059e6329b44c2f9c11\": container with ID starting with eed0f6b09096e25f5f6f030d62e8c2ac81d29ae259d583059e6329b44c2f9c11 not found: ID does not exist" containerID="eed0f6b09096e25f5f6f030d62e8c2ac81d29ae259d583059e6329b44c2f9c11"
Jul 27 17:41:26.239474 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:26.239415    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed0f6b09096e25f5f6f030d62e8c2ac81d29ae259d583059e6329b44c2f9c11"} err="failed to get container status \"eed0f6b09096e25f5f6f030d62e8c2ac81d29ae259d583059e6329b44c2f9c11\": rpc error: code = NotFound desc = could not find container \"eed0f6b09096e25f5f6f030d62e8c2ac81d29ae259d583059e6329b44c2f9c11\": container with ID starting with eed0f6b09096e25f5f6f030d62e8c2ac81d29ae259d583059e6329b44c2f9c11 not found: ID does not exist"
Jul 27 17:41:26.251956 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:26.251933    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-xr2fb"]
Jul 27 17:41:26.259313 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:26.259292    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-xr2fb"]
Jul 27 17:41:26.398438 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:26.398400    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe14455-6123-40c1-801b-5ff8888894a5" path="/var/lib/kubelet/pods/7fe14455-6123-40c1-801b-5ff8888894a5/volumes"
Jul 27 17:41:46.119140 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:46.119103    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6zkw2"]
Jul 27 17:41:46.119557 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:46.119281    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-6zkw2" podUID="9180da43-c5ef-4b79-b9d8-3a7b690d53b4" containerName="authorino" containerID="cri-o://e77fde90cf0df5663dcfd47bfdd5795a2d977c20303781475075ea6244789e04" gracePeriod=30
Jul 27 17:41:46.323588 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:46.323558    2571 generic.go:356] "Generic (PLEG): container finished" podID="9180da43-c5ef-4b79-b9d8-3a7b690d53b4" containerID="e77fde90cf0df5663dcfd47bfdd5795a2d977c20303781475075ea6244789e04" exitCode=0
Jul 27 17:41:46.323731 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:46.323594    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-6zkw2" event={"ID":"9180da43-c5ef-4b79-b9d8-3a7b690d53b4","Type":"ContainerDied","Data":"e77fde90cf0df5663dcfd47bfdd5795a2d977c20303781475075ea6244789e04"}
Jul 27 17:41:46.358215 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:46.358192    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-6zkw2"
Jul 27 17:41:46.414307 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:46.414247    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cztl4\" (UniqueName: \"kubernetes.io/projected/9180da43-c5ef-4b79-b9d8-3a7b690d53b4-kube-api-access-cztl4\") pod \"9180da43-c5ef-4b79-b9d8-3a7b690d53b4\" (UID: \"9180da43-c5ef-4b79-b9d8-3a7b690d53b4\") "
Jul 27 17:41:46.416150 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:46.416130    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9180da43-c5ef-4b79-b9d8-3a7b690d53b4-kube-api-access-cztl4" (OuterVolumeSpecName: "kube-api-access-cztl4") pod "9180da43-c5ef-4b79-b9d8-3a7b690d53b4" (UID: "9180da43-c5ef-4b79-b9d8-3a7b690d53b4"). InnerVolumeSpecName "kube-api-access-cztl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:41:46.515696 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:46.515671    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cztl4\" (UniqueName: \"kubernetes.io/projected/9180da43-c5ef-4b79-b9d8-3a7b690d53b4-kube-api-access-cztl4\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:41:47.329137 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:47.329104    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-6zkw2" event={"ID":"9180da43-c5ef-4b79-b9d8-3a7b690d53b4","Type":"ContainerDied","Data":"deb263231a4a763ef2058fe4803feb24512a5b0a90b70eb94ac457ad04ebe8ee"}
Jul 27 17:41:47.329584 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:47.329152    2571 scope.go:117] "RemoveContainer" containerID="e77fde90cf0df5663dcfd47bfdd5795a2d977c20303781475075ea6244789e04"
Jul 27 17:41:47.329584 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:47.329110    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-6zkw2"
Jul 27 17:41:47.350041 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:47.350012    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6zkw2"]
Jul 27 17:41:47.353454 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:47.353433    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-6zkw2"]
Jul 27 17:41:48.399314 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:48.399272    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9180da43-c5ef-4b79-b9d8-3a7b690d53b4" path="/var/lib/kubelet/pods/9180da43-c5ef-4b79-b9d8-3a7b690d53b4/volumes"
Jul 27 17:41:50.385123 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.385091    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["observability/jaeger-6bbd7d4c9d-z4ltk"]
Jul 27 17:41:50.385480 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.385414    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9180da43-c5ef-4b79-b9d8-3a7b690d53b4" containerName="authorino"
Jul 27 17:41:50.385480 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.385426    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9180da43-c5ef-4b79-b9d8-3a7b690d53b4" containerName="authorino"
Jul 27 17:41:50.385480 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.385453    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe14455-6123-40c1-801b-5ff8888894a5" containerName="authorino"
Jul 27 17:41:50.385480 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.385459    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe14455-6123-40c1-801b-5ff8888894a5" containerName="authorino"
Jul 27 17:41:50.385625 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.385523    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9180da43-c5ef-4b79-b9d8-3a7b690d53b4" containerName="authorino"
Jul 27 17:41:50.385625 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.385533    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe14455-6123-40c1-801b-5ff8888894a5" containerName="authorino"
Jul 27 17:41:50.389487 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.389470    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-z4ltk"
Jul 27 17:41:50.392144 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.392122    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"observability\"/\"jaeger-dockercfg-5hvhw\""
Jul 27 17:41:50.392490 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.392472    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"openshift-service-ca.crt\""
Jul 27 17:41:50.393211 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.393192    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"kube-root-ca.crt\""
Jul 27 17:41:50.402958 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.402937    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-z4ltk"]
Jul 27 17:41:50.446585 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.446560    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h28fw\" (UniqueName: \"kubernetes.io/projected/8c9aa433-c94a-4bef-9fb8-7b3059cea2fe-kube-api-access-h28fw\") pod \"jaeger-6bbd7d4c9d-z4ltk\" (UID: \"8c9aa433-c94a-4bef-9fb8-7b3059cea2fe\") " pod="observability/jaeger-6bbd7d4c9d-z4ltk"
Jul 27 17:41:50.547514 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.547460    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h28fw\" (UniqueName: \"kubernetes.io/projected/8c9aa433-c94a-4bef-9fb8-7b3059cea2fe-kube-api-access-h28fw\") pod \"jaeger-6bbd7d4c9d-z4ltk\" (UID: \"8c9aa433-c94a-4bef-9fb8-7b3059cea2fe\") " pod="observability/jaeger-6bbd7d4c9d-z4ltk"
Jul 27 17:41:50.558321 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.558287    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h28fw\" (UniqueName: \"kubernetes.io/projected/8c9aa433-c94a-4bef-9fb8-7b3059cea2fe-kube-api-access-h28fw\") pod \"jaeger-6bbd7d4c9d-z4ltk\" (UID: \"8c9aa433-c94a-4bef-9fb8-7b3059cea2fe\") " pod="observability/jaeger-6bbd7d4c9d-z4ltk"
Jul 27 17:41:50.698600 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.698525    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-z4ltk"
Jul 27 17:41:50.823606 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:50.823553    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-z4ltk"]
Jul 27 17:41:50.825663 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:41:50.825629    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c9aa433_c94a_4bef_9fb8_7b3059cea2fe.slice/crio-c4ca76e7a5af6da5c9ba584d8bac916c37eb32beb2448608fbb7a9dd34a96a8f WatchSource:0}: Error finding container c4ca76e7a5af6da5c9ba584d8bac916c37eb32beb2448608fbb7a9dd34a96a8f: Status 404 returned error can't find the container with id c4ca76e7a5af6da5c9ba584d8bac916c37eb32beb2448608fbb7a9dd34a96a8f
Jul 27 17:41:51.356128 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:51.356088    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-z4ltk" event={"ID":"8c9aa433-c94a-4bef-9fb8-7b3059cea2fe","Type":"ContainerStarted","Data":"c4ca76e7a5af6da5c9ba584d8bac916c37eb32beb2448608fbb7a9dd34a96a8f"}
Jul 27 17:41:53.367117 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:53.367080    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-z4ltk" event={"ID":"8c9aa433-c94a-4bef-9fb8-7b3059cea2fe","Type":"ContainerStarted","Data":"1e04b5c3a138d6029daa41b7aba43b402bd4962b57a6b73b0775aa338c1cee73"}
Jul 27 17:41:53.367546 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:53.367222    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="observability/jaeger-6bbd7d4c9d-z4ltk"
Jul 27 17:41:53.369102 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:53.369078    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="observability/jaeger-6bbd7d4c9d-z4ltk"
Jul 27 17:41:53.384269 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:53.384227    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="observability/jaeger-6bbd7d4c9d-z4ltk" podStartSLOduration=1.647956242 podStartE2EDuration="3.384211561s" totalImagesPullingTime="1.736255319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:41:53.383093637 +0000 UTC m=+795.427831735" watchObservedRunningTime="2026-07-27 17:41:53.384211561 +0000 UTC m=+795.428949642"
Jul 27 17:41:55.575377 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.575338    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-qz9gv"]
Jul 27 17:41:55.575865 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.575588    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-ff96d5959-qz9gv" podUID="c57478f8-0603-4e94-a5c2-c06a1babde8d" containerName="manager" containerID="cri-o://1f1016d68f04a30ec5a393f9eff98863e3df8b3fc1a1c5b13e2bf60d4b67211f" gracePeriod=10
Jul 27 17:41:55.600199 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.600176    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-nxnm8"]
Jul 27 17:41:55.608364 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.603651    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-nxnm8"
Jul 27 17:41:55.611238 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.611217    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-nxnm8"]
Jul 27 17:41:55.692191 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.692171    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3696de3b-7488-42d9-867e-962e7c2ddcd2-cert\") pod \"kserve-controller-manager-ff96d5959-nxnm8\" (UID: \"3696de3b-7488-42d9-867e-962e7c2ddcd2\") " pod="kserve/kserve-controller-manager-ff96d5959-nxnm8"
Jul 27 17:41:55.692273 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.692219    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvnpz\" (UniqueName: \"kubernetes.io/projected/3696de3b-7488-42d9-867e-962e7c2ddcd2-kube-api-access-mvnpz\") pod \"kserve-controller-manager-ff96d5959-nxnm8\" (UID: \"3696de3b-7488-42d9-867e-962e7c2ddcd2\") " pod="kserve/kserve-controller-manager-ff96d5959-nxnm8"
Jul 27 17:41:55.792870 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.792842    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnpz\" (UniqueName: \"kubernetes.io/projected/3696de3b-7488-42d9-867e-962e7c2ddcd2-kube-api-access-mvnpz\") pod \"kserve-controller-manager-ff96d5959-nxnm8\" (UID: \"3696de3b-7488-42d9-867e-962e7c2ddcd2\") " pod="kserve/kserve-controller-manager-ff96d5959-nxnm8"
Jul 27 17:41:55.793010 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.792910    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3696de3b-7488-42d9-867e-962e7c2ddcd2-cert\") pod \"kserve-controller-manager-ff96d5959-nxnm8\" (UID: \"3696de3b-7488-42d9-867e-962e7c2ddcd2\") " pod="kserve/kserve-controller-manager-ff96d5959-nxnm8"
Jul 27 17:41:55.795122 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.795100    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3696de3b-7488-42d9-867e-962e7c2ddcd2-cert\") pod \"kserve-controller-manager-ff96d5959-nxnm8\" (UID: \"3696de3b-7488-42d9-867e-962e7c2ddcd2\") " pod="kserve/kserve-controller-manager-ff96d5959-nxnm8"
Jul 27 17:41:55.802313 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.802286    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvnpz\" (UniqueName: \"kubernetes.io/projected/3696de3b-7488-42d9-867e-962e7c2ddcd2-kube-api-access-mvnpz\") pod \"kserve-controller-manager-ff96d5959-nxnm8\" (UID: \"3696de3b-7488-42d9-867e-962e7c2ddcd2\") " pod="kserve/kserve-controller-manager-ff96d5959-nxnm8"
Jul 27 17:41:55.821703 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.821685    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-qz9gv"
Jul 27 17:41:55.893946 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.893871    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbxrk\" (UniqueName: \"kubernetes.io/projected/c57478f8-0603-4e94-a5c2-c06a1babde8d-kube-api-access-tbxrk\") pod \"c57478f8-0603-4e94-a5c2-c06a1babde8d\" (UID: \"c57478f8-0603-4e94-a5c2-c06a1babde8d\") "
Jul 27 17:41:55.893946 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.893908    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c57478f8-0603-4e94-a5c2-c06a1babde8d-cert\") pod \"c57478f8-0603-4e94-a5c2-c06a1babde8d\" (UID: \"c57478f8-0603-4e94-a5c2-c06a1babde8d\") "
Jul 27 17:41:55.896066 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.896041    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57478f8-0603-4e94-a5c2-c06a1babde8d-cert" (OuterVolumeSpecName: "cert") pod "c57478f8-0603-4e94-a5c2-c06a1babde8d" (UID: "c57478f8-0603-4e94-a5c2-c06a1babde8d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:41:55.896170 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.896045    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57478f8-0603-4e94-a5c2-c06a1babde8d-kube-api-access-tbxrk" (OuterVolumeSpecName: "kube-api-access-tbxrk") pod "c57478f8-0603-4e94-a5c2-c06a1babde8d" (UID: "c57478f8-0603-4e94-a5c2-c06a1babde8d"). InnerVolumeSpecName "kube-api-access-tbxrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:41:55.952227 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.952196    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-nxnm8"
Jul 27 17:41:55.995105 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.995081    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbxrk\" (UniqueName: \"kubernetes.io/projected/c57478f8-0603-4e94-a5c2-c06a1babde8d-kube-api-access-tbxrk\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:41:55.995105 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:55.995104    2571 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c57478f8-0603-4e94-a5c2-c06a1babde8d-cert\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:41:56.105292 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:56.105217    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-nxnm8"]
Jul 27 17:41:56.107431 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:41:56.107393    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3696de3b_7488_42d9_867e_962e7c2ddcd2.slice/crio-c77133995ad38553f4763b799ded1e8039f1c8501129a1098458a50a20428104 WatchSource:0}: Error finding container c77133995ad38553f4763b799ded1e8039f1c8501129a1098458a50a20428104: Status 404 returned error can't find the container with id c77133995ad38553f4763b799ded1e8039f1c8501129a1098458a50a20428104
Jul 27 17:41:56.382533 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:56.382482    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-nxnm8" event={"ID":"3696de3b-7488-42d9-867e-962e7c2ddcd2","Type":"ContainerStarted","Data":"c77133995ad38553f4763b799ded1e8039f1c8501129a1098458a50a20428104"}
Jul 27 17:41:56.383709 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:56.383685    2571 generic.go:356] "Generic (PLEG): container finished" podID="c57478f8-0603-4e94-a5c2-c06a1babde8d" containerID="1f1016d68f04a30ec5a393f9eff98863e3df8b3fc1a1c5b13e2bf60d4b67211f" exitCode=0
Jul 27 17:41:56.383823 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:56.383726    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-qz9gv"
Jul 27 17:41:56.383823 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:56.383745    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-qz9gv" event={"ID":"c57478f8-0603-4e94-a5c2-c06a1babde8d","Type":"ContainerDied","Data":"1f1016d68f04a30ec5a393f9eff98863e3df8b3fc1a1c5b13e2bf60d4b67211f"}
Jul 27 17:41:56.383823 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:56.383780    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-qz9gv" event={"ID":"c57478f8-0603-4e94-a5c2-c06a1babde8d","Type":"ContainerDied","Data":"e7e07a80c744afdea9fa188a07e4bd9d6c8bbe1e7ba2fa426acf923f423fe92f"}
Jul 27 17:41:56.383823 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:56.383815    2571 scope.go:117] "RemoveContainer" containerID="1f1016d68f04a30ec5a393f9eff98863e3df8b3fc1a1c5b13e2bf60d4b67211f"
Jul 27 17:41:56.392139 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:56.392123    2571 scope.go:117] "RemoveContainer" containerID="1f1016d68f04a30ec5a393f9eff98863e3df8b3fc1a1c5b13e2bf60d4b67211f"
Jul 27 17:41:56.392388 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:41:56.392369    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1016d68f04a30ec5a393f9eff98863e3df8b3fc1a1c5b13e2bf60d4b67211f\": container with ID starting with 1f1016d68f04a30ec5a393f9eff98863e3df8b3fc1a1c5b13e2bf60d4b67211f not found: ID does not exist" containerID="1f1016d68f04a30ec5a393f9eff98863e3df8b3fc1a1c5b13e2bf60d4b67211f"
Jul 27 17:41:56.392455 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:56.392400    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1016d68f04a30ec5a393f9eff98863e3df8b3fc1a1c5b13e2bf60d4b67211f"} err="failed to get container status \"1f1016d68f04a30ec5a393f9eff98863e3df8b3fc1a1c5b13e2bf60d4b67211f\": rpc error: code = NotFound desc = could not find container \"1f1016d68f04a30ec5a393f9eff98863e3df8b3fc1a1c5b13e2bf60d4b67211f\": container with ID starting with 1f1016d68f04a30ec5a393f9eff98863e3df8b3fc1a1c5b13e2bf60d4b67211f not found: ID does not exist"
Jul 27 17:41:56.405178 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:56.405158    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-qz9gv"]
Jul 27 17:41:56.410244 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:56.410225    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-qz9gv"]
Jul 27 17:41:58.398768 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:58.398735    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c57478f8-0603-4e94-a5c2-c06a1babde8d" path="/var/lib/kubelet/pods/c57478f8-0603-4e94-a5c2-c06a1babde8d/volumes"
Jul 27 17:41:59.405262 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:59.405222    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-nxnm8" event={"ID":"3696de3b-7488-42d9-867e-962e7c2ddcd2","Type":"ContainerStarted","Data":"b1f6b60ab6a37643cf5aa6f6fad4b57480e3d87bde4cbd9759d1f93d45973cc4"}
Jul 27 17:41:59.405759 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:41:59.405356    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-nxnm8"
Jul 27 17:42:30.414957 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:30.414925    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-nxnm8"
Jul 27 17:42:30.435455 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:30.435405    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-nxnm8" podStartSLOduration=32.269293648 podStartE2EDuration="35.435391003s" totalImagesPullingTime="3.166097355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:41:59.420609599 +0000 UTC m=+801.465347680" watchObservedRunningTime="2026-07-27 17:42:30.435391003 +0000 UTC m=+832.480129084"
Jul 27 17:42:31.563350 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.563321    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-5r24t"]
Jul 27 17:42:31.563743 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.563619    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57478f8-0603-4e94-a5c2-c06a1babde8d" containerName="manager"
Jul 27 17:42:31.563743 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.563631    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57478f8-0603-4e94-a5c2-c06a1babde8d" containerName="manager"
Jul 27 17:42:31.563743 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.563691    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57478f8-0603-4e94-a5c2-c06a1babde8d" containerName="manager"
Jul 27 17:42:31.567977 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.567959    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-5r24t"
Jul 27 17:42:31.570593 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.570570    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 27 17:42:31.570593 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.570587    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-2zmnv\""
Jul 27 17:42:31.576359 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.576340    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-5r24t"]
Jul 27 17:42:31.589589 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.589564    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdg79\" (UniqueName: \"kubernetes.io/projected/04f23002-9c6c-4eba-8977-f0e43b81f922-kube-api-access-pdg79\") pod \"model-serving-api-7b954ffc95-5r24t\" (UID: \"04f23002-9c6c-4eba-8977-f0e43b81f922\") " pod="kserve/model-serving-api-7b954ffc95-5r24t"
Jul 27 17:42:31.589713 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.589624    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f23002-9c6c-4eba-8977-f0e43b81f922-tls-certs\") pod \"model-serving-api-7b954ffc95-5r24t\" (UID: \"04f23002-9c6c-4eba-8977-f0e43b81f922\") " pod="kserve/model-serving-api-7b954ffc95-5r24t"
Jul 27 17:42:31.690698 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.690660    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f23002-9c6c-4eba-8977-f0e43b81f922-tls-certs\") pod \"model-serving-api-7b954ffc95-5r24t\" (UID: \"04f23002-9c6c-4eba-8977-f0e43b81f922\") " pod="kserve/model-serving-api-7b954ffc95-5r24t"
Jul 27 17:42:31.690896 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.690711    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdg79\" (UniqueName: \"kubernetes.io/projected/04f23002-9c6c-4eba-8977-f0e43b81f922-kube-api-access-pdg79\") pod \"model-serving-api-7b954ffc95-5r24t\" (UID: \"04f23002-9c6c-4eba-8977-f0e43b81f922\") " pod="kserve/model-serving-api-7b954ffc95-5r24t"
Jul 27 17:42:31.693164 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.693137    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f23002-9c6c-4eba-8977-f0e43b81f922-tls-certs\") pod \"model-serving-api-7b954ffc95-5r24t\" (UID: \"04f23002-9c6c-4eba-8977-f0e43b81f922\") " pod="kserve/model-serving-api-7b954ffc95-5r24t"
Jul 27 17:42:31.699974 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.699951    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdg79\" (UniqueName: \"kubernetes.io/projected/04f23002-9c6c-4eba-8977-f0e43b81f922-kube-api-access-pdg79\") pod \"model-serving-api-7b954ffc95-5r24t\" (UID: \"04f23002-9c6c-4eba-8977-f0e43b81f922\") " pod="kserve/model-serving-api-7b954ffc95-5r24t"
Jul 27 17:42:31.879248 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:31.879159    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-5r24t"
Jul 27 17:42:32.004950 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:32.004927    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-5r24t"]
Jul 27 17:42:32.007462 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:42:32.007438    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f23002_9c6c_4eba_8977_f0e43b81f922.slice/crio-9c275e53a676633a1920ed70c28ba1b7d5934a0cf1089a8d1d7702daceb539c5 WatchSource:0}: Error finding container 9c275e53a676633a1920ed70c28ba1b7d5934a0cf1089a8d1d7702daceb539c5: Status 404 returned error can't find the container with id 9c275e53a676633a1920ed70c28ba1b7d5934a0cf1089a8d1d7702daceb539c5
Jul 27 17:42:32.553090 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:32.553048    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-5r24t" event={"ID":"04f23002-9c6c-4eba-8977-f0e43b81f922","Type":"ContainerStarted","Data":"9c275e53a676633a1920ed70c28ba1b7d5934a0cf1089a8d1d7702daceb539c5"}
Jul 27 17:42:34.565537 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:34.565472    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-5r24t" event={"ID":"04f23002-9c6c-4eba-8977-f0e43b81f922","Type":"ContainerStarted","Data":"252ae4e12d98ad90787033445d02f0f5caaf4188bc648fbe1afb8e03e37dc53d"}
Jul 27 17:42:34.565924 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:34.565554    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-5r24t"
Jul 27 17:42:34.583758 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:34.583707    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-5r24t" podStartSLOduration=1.587944834 podStartE2EDuration="3.583694702s" totalImagesPullingTime="1.995749868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:42:34.582362189 +0000 UTC m=+836.627100270" watchObservedRunningTime="2026-07-27 17:42:34.583694702 +0000 UTC m=+836.628432828"
Jul 27 17:42:45.574217 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:45.574187    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-5r24t"
Jul 27 17:42:47.191602 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:47.191563    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-zggbs"]
Jul 27 17:42:47.196277 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:47.196261    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zggbs"
Jul 27 17:42:47.203464 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:47.203442    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zggbs"]
Jul 27 17:42:47.301573 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:47.301532    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrvvc\" (UniqueName: \"kubernetes.io/projected/c33e4dc0-1d69-435e-96aa-189134b3d1ab-kube-api-access-lrvvc\") pod \"s3-init-zggbs\" (UID: \"c33e4dc0-1d69-435e-96aa-189134b3d1ab\") " pod="kserve/s3-init-zggbs"
Jul 27 17:42:47.301731 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:47.301608    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33e4dc0-1d69-435e-96aa-189134b3d1ab-model-cache\") pod \"s3-init-zggbs\" (UID: \"c33e4dc0-1d69-435e-96aa-189134b3d1ab\") " pod="kserve/s3-init-zggbs"
Jul 27 17:42:47.401896 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:47.401861    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvvc\" (UniqueName: \"kubernetes.io/projected/c33e4dc0-1d69-435e-96aa-189134b3d1ab-kube-api-access-lrvvc\") pod \"s3-init-zggbs\" (UID: \"c33e4dc0-1d69-435e-96aa-189134b3d1ab\") " pod="kserve/s3-init-zggbs"
Jul 27 17:42:47.402092 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:47.401932    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33e4dc0-1d69-435e-96aa-189134b3d1ab-model-cache\") pod \"s3-init-zggbs\" (UID: \"c33e4dc0-1d69-435e-96aa-189134b3d1ab\") " pod="kserve/s3-init-zggbs"
Jul 27 17:42:47.402261 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:47.402238    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33e4dc0-1d69-435e-96aa-189134b3d1ab-model-cache\") pod \"s3-init-zggbs\" (UID: \"c33e4dc0-1d69-435e-96aa-189134b3d1ab\") " pod="kserve/s3-init-zggbs"
Jul 27 17:42:47.410858 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:47.410841    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvvc\" (UniqueName: \"kubernetes.io/projected/c33e4dc0-1d69-435e-96aa-189134b3d1ab-kube-api-access-lrvvc\") pod \"s3-init-zggbs\" (UID: \"c33e4dc0-1d69-435e-96aa-189134b3d1ab\") " pod="kserve/s3-init-zggbs"
Jul 27 17:42:47.506352 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:47.506320    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zggbs"
Jul 27 17:42:47.633367 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:47.633347    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zggbs"]
Jul 27 17:42:47.635624 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:42:47.635601    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc33e4dc0_1d69_435e_96aa_189134b3d1ab.slice/crio-11275c3168898c8f69f9ba2945a2aa64b36867ecb370a4283548ece7658e9b8d WatchSource:0}: Error finding container 11275c3168898c8f69f9ba2945a2aa64b36867ecb370a4283548ece7658e9b8d: Status 404 returned error can't find the container with id 11275c3168898c8f69f9ba2945a2aa64b36867ecb370a4283548ece7658e9b8d
Jul 27 17:42:48.632026 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:48.631996    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zggbs" event={"ID":"c33e4dc0-1d69-435e-96aa-189134b3d1ab","Type":"ContainerStarted","Data":"11275c3168898c8f69f9ba2945a2aa64b36867ecb370a4283548ece7658e9b8d"}
Jul 27 17:42:53.658620 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:53.658530    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zggbs" event={"ID":"c33e4dc0-1d69-435e-96aa-189134b3d1ab","Type":"ContainerStarted","Data":"a58cafc85a739844c5eb97cd94cf9bb2cb4ae24e29a1403be0493c33e0d39be6"}
Jul 27 17:42:55.668692 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:55.668657    2571 generic.go:356] "Generic (PLEG): container finished" podID="c33e4dc0-1d69-435e-96aa-189134b3d1ab" containerID="a58cafc85a739844c5eb97cd94cf9bb2cb4ae24e29a1403be0493c33e0d39be6" exitCode=0
Jul 27 17:42:55.669054 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:42:55.668728    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zggbs" event={"ID":"c33e4dc0-1d69-435e-96aa-189134b3d1ab","Type":"ContainerDied","Data":"a58cafc85a739844c5eb97cd94cf9bb2cb4ae24e29a1403be0493c33e0d39be6"}
Jul 27 17:43:00.696564 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:43:00.696472    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zggbs" event={"ID":"c33e4dc0-1d69-435e-96aa-189134b3d1ab","Type":"ContainerStarted","Data":"7f52fc683669c9a74a250a1fa7b0ded87ec4d71d1cf11168df3c142b2a264f59"}
Jul 27 17:43:00.712790 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:43:00.712739    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-zggbs" podStartSLOduration=1.31607953 podStartE2EDuration="13.712720691s" totalImagesPullingTime="10.228680983s" totalInitContainerRuntime="2.167960178s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:42:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:43:00.712518072 +0000 UTC m=+862.757256154" watchObservedRunningTime="2026-07-27 17:43:00.712720691 +0000 UTC m=+862.757458775"
Jul 27 17:44:06.016145 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:06.016114    2571 generic.go:356] "Generic (PLEG): container finished" podID="c33e4dc0-1d69-435e-96aa-189134b3d1ab" containerID="7f52fc683669c9a74a250a1fa7b0ded87ec4d71d1cf11168df3c142b2a264f59" exitCode=0
Jul 27 17:44:06.016594 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:06.016197    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zggbs" event={"ID":"c33e4dc0-1d69-435e-96aa-189134b3d1ab","Type":"ContainerDied","Data":"7f52fc683669c9a74a250a1fa7b0ded87ec4d71d1cf11168df3c142b2a264f59"}
Jul 27 17:44:07.148246 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:07.148221    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zggbs"
Jul 27 17:44:07.255056 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:07.255007    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33e4dc0-1d69-435e-96aa-189134b3d1ab-model-cache\") pod \"c33e4dc0-1d69-435e-96aa-189134b3d1ab\" (UID: \"c33e4dc0-1d69-435e-96aa-189134b3d1ab\") "
Jul 27 17:44:07.255240 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:07.255071    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrvvc\" (UniqueName: \"kubernetes.io/projected/c33e4dc0-1d69-435e-96aa-189134b3d1ab-kube-api-access-lrvvc\") pod \"c33e4dc0-1d69-435e-96aa-189134b3d1ab\" (UID: \"c33e4dc0-1d69-435e-96aa-189134b3d1ab\") "
Jul 27 17:44:07.257187 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:07.257158    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33e4dc0-1d69-435e-96aa-189134b3d1ab-kube-api-access-lrvvc" (OuterVolumeSpecName: "kube-api-access-lrvvc") pod "c33e4dc0-1d69-435e-96aa-189134b3d1ab" (UID: "c33e4dc0-1d69-435e-96aa-189134b3d1ab"). InnerVolumeSpecName "kube-api-access-lrvvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:44:07.274296 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:07.274238    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33e4dc0-1d69-435e-96aa-189134b3d1ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "c33e4dc0-1d69-435e-96aa-189134b3d1ab" (UID: "c33e4dc0-1d69-435e-96aa-189134b3d1ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:44:07.356197 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:07.356164    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33e4dc0-1d69-435e-96aa-189134b3d1ab-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:44:07.356197 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:07.356193    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrvvc\" (UniqueName: \"kubernetes.io/projected/c33e4dc0-1d69-435e-96aa-189134b3d1ab-kube-api-access-lrvvc\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:44:08.028076 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:08.028036    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zggbs" event={"ID":"c33e4dc0-1d69-435e-96aa-189134b3d1ab","Type":"ContainerDied","Data":"11275c3168898c8f69f9ba2945a2aa64b36867ecb370a4283548ece7658e9b8d"}
Jul 27 17:44:08.028076 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:08.028074    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11275c3168898c8f69f9ba2945a2aa64b36867ecb370a4283548ece7658e9b8d"
Jul 27 17:44:08.028076 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:08.028050    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zggbs"
Jul 27 17:44:30.384724 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.384688    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"]
Jul 27 17:44:30.385164 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.385007    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c33e4dc0-1d69-435e-96aa-189134b3d1ab" containerName="s3-init"
Jul 27 17:44:30.385164 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.385018    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c33e4dc0-1d69-435e-96aa-189134b3d1ab" containerName="s3-init"
Jul 27 17:44:30.385164 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.385039    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c33e4dc0-1d69-435e-96aa-189134b3d1ab" containerName="download-hf-model"
Jul 27 17:44:30.385164 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.385045    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c33e4dc0-1d69-435e-96aa-189134b3d1ab" containerName="download-hf-model"
Jul 27 17:44:30.385164 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.385092    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c33e4dc0-1d69-435e-96aa-189134b3d1ab" containerName="s3-init"
Jul 27 17:44:30.388322 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.388306    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.391244 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.391218    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Jul 27 17:44:30.391363 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.391321    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 27 17:44:30.391415 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.391368    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-s984t\""
Jul 27 17:44:30.391451 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.391427    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Jul 27 17:44:30.391570 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.391553    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Jul 27 17:44:30.400035 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.400016    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"]
Jul 27 17:44:30.546222 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.546190    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.546222 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.546227    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.546452 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.546250    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.546452 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.546339    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.546452 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.546376    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.546452 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.546396    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9d99\" (UniqueName: \"kubernetes.io/projected/e21648df-e6e0-4a36-9e2d-986d42f69393-kube-api-access-p9d99\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.546452 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.546437    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21648df-e6e0-4a36-9e2d-986d42f69393-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.647665 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.647579    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.647665 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.647620    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.647665 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.647642    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9d99\" (UniqueName: \"kubernetes.io/projected/e21648df-e6e0-4a36-9e2d-986d42f69393-kube-api-access-p9d99\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.647665 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.647667    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21648df-e6e0-4a36-9e2d-986d42f69393-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.648000 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.647736    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.648000 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.647771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.648000 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.647799    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.648130 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.648037    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.648193 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.648151    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.648193 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.648160    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.648278 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.648209    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.649961 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.649939    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.650212 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.650196    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21648df-e6e0-4a36-9e2d-986d42f69393-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.659067 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.659034    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9d99\" (UniqueName: \"kubernetes.io/projected/e21648df-e6e0-4a36-9e2d-986d42f69393-kube-api-access-p9d99\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.700086 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.700055    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:44:30.838210 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:30.838182    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"]
Jul 27 17:44:30.840007 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:44:30.839979    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21648df_e6e0_4a36_9e2d_986d42f69393.slice/crio-2b56277cb68bb910b33473beafc1bb85b5fe6989fd5c9df8f9b23a29e4dfb056 WatchSource:0}: Error finding container 2b56277cb68bb910b33473beafc1bb85b5fe6989fd5c9df8f9b23a29e4dfb056: Status 404 returned error can't find the container with id 2b56277cb68bb910b33473beafc1bb85b5fe6989fd5c9df8f9b23a29e4dfb056
Jul 27 17:44:31.146698 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:31.146661    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" event={"ID":"e21648df-e6e0-4a36-9e2d-986d42f69393","Type":"ContainerStarted","Data":"4e6e64236ed82201ccdaaba7a58dd66508fda18e561e843e042505f57e4eb86e"}
Jul 27 17:44:31.146866 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:31.146726    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" event={"ID":"e21648df-e6e0-4a36-9e2d-986d42f69393","Type":"ContainerStarted","Data":"2b56277cb68bb910b33473beafc1bb85b5fe6989fd5c9df8f9b23a29e4dfb056"}
Jul 27 17:44:39.191247 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:39.191161    2571 generic.go:356] "Generic (PLEG): container finished" podID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerID="4e6e64236ed82201ccdaaba7a58dd66508fda18e561e843e042505f57e4eb86e" exitCode=0
Jul 27 17:44:39.191624 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:39.191174    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" event={"ID":"e21648df-e6e0-4a36-9e2d-986d42f69393","Type":"ContainerDied","Data":"4e6e64236ed82201ccdaaba7a58dd66508fda18e561e843e042505f57e4eb86e"}
Jul 27 17:44:40.204941 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:40.204849    2571 generic.go:356] "Generic (PLEG): container finished" podID="e27bde2a-4e4d-4ea9-9b27-4665b50a751c" containerID="03dc198d63c94eec0aeef5045981412c47bc18d175f8504d984f85799d4c3f2c" exitCode=0
Jul 27 17:44:40.205653 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:40.205017    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb" event={"ID":"e27bde2a-4e4d-4ea9-9b27-4665b50a751c","Type":"ContainerDied","Data":"03dc198d63c94eec0aeef5045981412c47bc18d175f8504d984f85799d4c3f2c"}
Jul 27 17:44:40.205653 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:40.205546    2571 scope.go:117] "RemoveContainer" containerID="03dc198d63c94eec0aeef5045981412c47bc18d175f8504d984f85799d4c3f2c"
Jul 27 17:44:40.207239 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:40.207201    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:44:40.207341 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:40.207293    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:44:40.207415 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:40.207387    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:44:40.842631 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:40.842591    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:44:41.239923 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:41.239842    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb" event={"ID":"e27bde2a-4e4d-4ea9-9b27-4665b50a751c","Type":"ContainerStarted","Data":"f59e100aa7d143f973a3f86b64a05827ee2b17481f09aa98066c65f4cc594783"}
Jul 27 17:44:41.291086 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:41.291048    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:44:41.314562 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:41.292318    2571 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-spfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Jul 27 17:44:41.314562 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:41.292375    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb" podUID="e27bde2a-4e4d-4ea9-9b27-4665b50a751c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 27 17:44:42.291610 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:42.291568    2571 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-spfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Jul 27 17:44:42.292029 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:42.291638    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb" podUID="e27bde2a-4e4d-4ea9-9b27-4665b50a751c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 27 17:44:43.291097 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:43.291056    2571 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-spfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Jul 27 17:44:43.291295 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:43.291125    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb" podUID="e27bde2a-4e4d-4ea9-9b27-4665b50a751c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 27 17:44:44.291455 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:44.291416    2571 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-spfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Jul 27 17:44:44.291981 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:44.291552    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb" podUID="e27bde2a-4e4d-4ea9-9b27-4665b50a751c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 27 17:44:45.290480 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:45.290444    2571 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-spfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Jul 27 17:44:45.290671 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:45.290517    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb" podUID="e27bde2a-4e4d-4ea9-9b27-4665b50a751c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 27 17:44:46.291117 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:46.290903    2571 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-spfsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Jul 27 17:44:46.291117 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:46.290964    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb" podUID="e27bde2a-4e4d-4ea9-9b27-4665b50a751c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 27 17:44:47.296047 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:47.296015    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:44:47.296530 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:47.296449    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:44:47.297184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:44:47.297159    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-spfsb"
Jul 27 17:45:10.422471 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:45:10.422426    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" event={"ID":"e21648df-e6e0-4a36-9e2d-986d42f69393","Type":"ContainerStarted","Data":"bddb60878ba1432a42cdecdd38e6d018996956d2dc928898be092a3c21cfc722"}
Jul 27 17:45:10.448403 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:45:10.448351    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podStartSLOduration=2.159885923 podStartE2EDuration="40.448336295s" totalImagesPullingTime="30.474009609s" totalInitContainerRuntime="7.814440763s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:45:10.445165768 +0000 UTC m=+992.489903889" watchObservedRunningTime="2026-07-27 17:45:10.448336295 +0000 UTC m=+992.493074377"
Jul 27 17:45:10.700866 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:45:10.700781    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:45:10.700866 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:45:10.700850    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:45:10.702621 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:45:10.702578    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 27 17:45:20.701339 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:45:20.701296    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 27 17:45:30.701103 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:45:30.701054    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 27 17:45:40.701260 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:45:40.701213    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 27 17:45:50.700675 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:45:50.700630    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 27 17:46:00.700721 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:46:00.700682    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 27 17:46:10.701461 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:46:10.701410    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 27 17:46:20.701112 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:46:20.701064    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 27 17:46:30.701384 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:46:30.701274    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 27 17:46:40.700873 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:46:40.700817    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 27 17:46:50.701070 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:46:50.701026    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 27 17:47:00.710057 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:00.710023    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:47:00.718300 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:00.718278    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:47:03.898348 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:03.898305    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"]
Jul 27 17:47:03.903989 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:03.903959    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:03.908140 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:03.908116    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Jul 27 17:47:03.909582 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:03.908490    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 27 17:47:03.909582 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:03.909246    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 27 17:47:03.909582 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:03.909537    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-s9z8k\""
Jul 27 17:47:03.918209 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:03.917782    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"]
Jul 27 17:47:04.087418 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.087372    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c740fca8-20c7-4d1c-ab5e-966a0d057146-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.087418 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.087418    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.087718 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.087434    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trgj8\" (UniqueName: \"kubernetes.io/projected/c740fca8-20c7-4d1c-ab5e-966a0d057146-kube-api-access-trgj8\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.087718 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.087451    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.087718 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.087535    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.087718 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.087561    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.087718 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.087585    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.087718 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.087678    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.088010 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.087733    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.188904 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.188799    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.188904 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.188834    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trgj8\" (UniqueName: \"kubernetes.io/projected/c740fca8-20c7-4d1c-ab5e-966a0d057146-kube-api-access-trgj8\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.188904 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.188853    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.188904 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.188874    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.189253 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.188993    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.189253 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.189037    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.189253 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.189114    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.189253 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.189157    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.189253 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.189212    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c740fca8-20c7-4d1c-ab5e-966a0d057146-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.189253 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.189249    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.189583 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.189416    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.189583 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.189462    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.189696 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.189616    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.189948 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.189927    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c740fca8-20c7-4d1c-ab5e-966a0d057146-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.191375 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.191350    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.191473 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.191456    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.197114 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.197095    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.197257 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.197240    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trgj8\" (UniqueName: \"kubernetes.io/projected/c740fca8-20c7-4d1c-ab5e-966a0d057146-kube-api-access-trgj8\") pod \"router-gateway-1-openshift-default-66bbccc5fb-88dcs\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.226387 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.226361    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:04.361407 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.361386    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"]
Jul 27 17:47:04.364656 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:47:04.364624    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc740fca8_20c7_4d1c_ab5e_966a0d057146.slice/crio-134537223f9106fdc665042d7ea0dc7b11197eb57630c5fdd3aa028812ebbbf0 WatchSource:0}: Error finding container 134537223f9106fdc665042d7ea0dc7b11197eb57630c5fdd3aa028812ebbbf0: Status 404 returned error can't find the container with id 134537223f9106fdc665042d7ea0dc7b11197eb57630c5fdd3aa028812ebbbf0
Jul 27 17:47:04.367130 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.367093    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:47:04.367211 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.367176    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:47:04.367250 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:04.367213    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:47:05.020564 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:05.020530    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs" event={"ID":"c740fca8-20c7-4d1c-ab5e-966a0d057146","Type":"ContainerStarted","Data":"eaad7593a234dde2b99a79bc5029a9d820694f61c9f4e6a113415636ab7f3521"}
Jul 27 17:47:05.021202 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:05.020595    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs" event={"ID":"c740fca8-20c7-4d1c-ab5e-966a0d057146","Type":"ContainerStarted","Data":"134537223f9106fdc665042d7ea0dc7b11197eb57630c5fdd3aa028812ebbbf0"}
Jul 27 17:47:05.043061 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:05.043008    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs" podStartSLOduration=2.042993964 podStartE2EDuration="2.042993964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:47:05.041041808 +0000 UTC m=+1107.085779889" watchObservedRunningTime="2026-07-27 17:47:05.042993964 +0000 UTC m=+1107.087732044"
Jul 27 17:47:05.227448 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:05.227413    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:05.232615 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:05.232588    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:06.026769 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:06.026739    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:06.027887 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:06.027863    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:25.233135 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:25.233100    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"]
Jul 27 17:47:30.114062 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:30.114020    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"]
Jul 27 17:47:30.114512 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:30.114341    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs" podUID="c740fca8-20c7-4d1c-ab5e-966a0d057146" containerName="istio-proxy" containerID="cri-o://eaad7593a234dde2b99a79bc5029a9d820694f61c9f4e6a113415636ab7f3521" gracePeriod=30
Jul 27 17:47:33.155982 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.155959    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:33.166374 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.166350    2571 generic.go:356] "Generic (PLEG): container finished" podID="c740fca8-20c7-4d1c-ab5e-966a0d057146" containerID="eaad7593a234dde2b99a79bc5029a9d820694f61c9f4e6a113415636ab7f3521" exitCode=0
Jul 27 17:47:33.166484 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.166386    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"
Jul 27 17:47:33.166484 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.166429    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs" event={"ID":"c740fca8-20c7-4d1c-ab5e-966a0d057146","Type":"ContainerDied","Data":"eaad7593a234dde2b99a79bc5029a9d820694f61c9f4e6a113415636ab7f3521"}
Jul 27 17:47:33.166484 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.166480    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs" event={"ID":"c740fca8-20c7-4d1c-ab5e-966a0d057146","Type":"ContainerDied","Data":"134537223f9106fdc665042d7ea0dc7b11197eb57630c5fdd3aa028812ebbbf0"}
Jul 27 17:47:33.166611 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.166515    2571 scope.go:117] "RemoveContainer" containerID="eaad7593a234dde2b99a79bc5029a9d820694f61c9f4e6a113415636ab7f3521"
Jul 27 17:47:33.174601 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.174586    2571 scope.go:117] "RemoveContainer" containerID="eaad7593a234dde2b99a79bc5029a9d820694f61c9f4e6a113415636ab7f3521"
Jul 27 17:47:33.174825 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:47:33.174809    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaad7593a234dde2b99a79bc5029a9d820694f61c9f4e6a113415636ab7f3521\": container with ID starting with eaad7593a234dde2b99a79bc5029a9d820694f61c9f4e6a113415636ab7f3521 not found: ID does not exist" containerID="eaad7593a234dde2b99a79bc5029a9d820694f61c9f4e6a113415636ab7f3521"
Jul 27 17:47:33.174871 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.174839    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaad7593a234dde2b99a79bc5029a9d820694f61c9f4e6a113415636ab7f3521"} err="failed to get container status \"eaad7593a234dde2b99a79bc5029a9d820694f61c9f4e6a113415636ab7f3521\": rpc error: code = NotFound desc = could not find container \"eaad7593a234dde2b99a79bc5029a9d820694f61c9f4e6a113415636ab7f3521\": container with ID starting with eaad7593a234dde2b99a79bc5029a9d820694f61c9f4e6a113415636ab7f3521 not found: ID does not exist"
Jul 27 17:47:33.226999 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.226970    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-envoy\") pod \"c740fca8-20c7-4d1c-ab5e-966a0d057146\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") "
Jul 27 17:47:33.227184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.227007    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-token\") pod \"c740fca8-20c7-4d1c-ab5e-966a0d057146\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") "
Jul 27 17:47:33.227184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.227070    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-podinfo\") pod \"c740fca8-20c7-4d1c-ab5e-966a0d057146\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") "
Jul 27 17:47:33.227184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.227102    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-data\") pod \"c740fca8-20c7-4d1c-ab5e-966a0d057146\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") "
Jul 27 17:47:33.227184 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.227172    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-workload-certs\") pod \"c740fca8-20c7-4d1c-ab5e-966a0d057146\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") "
Jul 27 17:47:33.227399 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.227221    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c740fca8-20c7-4d1c-ab5e-966a0d057146-istiod-ca-cert\") pod \"c740fca8-20c7-4d1c-ab5e-966a0d057146\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") "
Jul 27 17:47:33.227399 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.227276    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-workload-socket\") pod \"c740fca8-20c7-4d1c-ab5e-966a0d057146\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") "
Jul 27 17:47:33.227399 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.227323    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trgj8\" (UniqueName: \"kubernetes.io/projected/c740fca8-20c7-4d1c-ab5e-966a0d057146-kube-api-access-trgj8\") pod \"c740fca8-20c7-4d1c-ab5e-966a0d057146\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") "
Jul 27 17:47:33.227399 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.227324    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-data" (OuterVolumeSpecName: "istio-data") pod "c740fca8-20c7-4d1c-ab5e-966a0d057146" (UID: "c740fca8-20c7-4d1c-ab5e-966a0d057146"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:33.227399 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.227353    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-credential-socket\") pod \"c740fca8-20c7-4d1c-ab5e-966a0d057146\" (UID: \"c740fca8-20c7-4d1c-ab5e-966a0d057146\") "
Jul 27 17:47:33.227690 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.227648    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-data\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:33.227846 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.227794    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c740fca8-20c7-4d1c-ab5e-966a0d057146" (UID: "c740fca8-20c7-4d1c-ab5e-966a0d057146"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:33.228039 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.227854    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c740fca8-20c7-4d1c-ab5e-966a0d057146" (UID: "c740fca8-20c7-4d1c-ab5e-966a0d057146"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:33.228039 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.227979    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c740fca8-20c7-4d1c-ab5e-966a0d057146" (UID: "c740fca8-20c7-4d1c-ab5e-966a0d057146"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:33.228179 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.228147    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c740fca8-20c7-4d1c-ab5e-966a0d057146-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c740fca8-20c7-4d1c-ab5e-966a0d057146" (UID: "c740fca8-20c7-4d1c-ab5e-966a0d057146"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:47:33.229597 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.229558    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c740fca8-20c7-4d1c-ab5e-966a0d057146" (UID: "c740fca8-20c7-4d1c-ab5e-966a0d057146"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:33.229759 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.229632    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-token" (OuterVolumeSpecName: "istio-token") pod "c740fca8-20c7-4d1c-ab5e-966a0d057146" (UID: "c740fca8-20c7-4d1c-ab5e-966a0d057146"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:47:33.229759 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.229730    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c740fca8-20c7-4d1c-ab5e-966a0d057146-kube-api-access-trgj8" (OuterVolumeSpecName: "kube-api-access-trgj8") pod "c740fca8-20c7-4d1c-ab5e-966a0d057146" (UID: "c740fca8-20c7-4d1c-ab5e-966a0d057146"). InnerVolumeSpecName "kube-api-access-trgj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:47:33.229918 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.229896    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c740fca8-20c7-4d1c-ab5e-966a0d057146" (UID: "c740fca8-20c7-4d1c-ab5e-966a0d057146"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 17:47:33.328541 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.328482    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-credential-socket\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:33.328541 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.328538    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-envoy\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:33.328541 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.328551    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-token\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:33.328760 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.328560    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c740fca8-20c7-4d1c-ab5e-966a0d057146-istio-podinfo\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:33.328760 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.328568    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-workload-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:33.328760 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.328575    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c740fca8-20c7-4d1c-ab5e-966a0d057146-istiod-ca-cert\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:33.328760 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.328583    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c740fca8-20c7-4d1c-ab5e-966a0d057146-workload-socket\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:33.328760 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.328590    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trgj8\" (UniqueName: \"kubernetes.io/projected/c740fca8-20c7-4d1c-ab5e-966a0d057146-kube-api-access-trgj8\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:33.489160 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.489133    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"]
Jul 27 17:47:33.494976 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:33.494948    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-88dcs"]
Jul 27 17:47:34.398899 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:34.398859    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c740fca8-20c7-4d1c-ab5e-966a0d057146" path="/var/lib/kubelet/pods/c740fca8-20c7-4d1c-ab5e-966a0d057146/volumes"
Jul 27 17:47:40.254868 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.254828    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main" containerID="cri-o://bddb60878ba1432a42cdecdd38e6d018996956d2dc928898be092a3c21cfc722" gracePeriod=45
Jul 27 17:47:40.710751 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.710701    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 27 17:47:40.921567 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.921542    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:47:40.992695 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.992620    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9d99\" (UniqueName: \"kubernetes.io/projected/e21648df-e6e0-4a36-9e2d-986d42f69393-kube-api-access-p9d99\") pod \"e21648df-e6e0-4a36-9e2d-986d42f69393\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") "
Jul 27 17:47:40.992695 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.992653    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21648df-e6e0-4a36-9e2d-986d42f69393-tls-certs\") pod \"e21648df-e6e0-4a36-9e2d-986d42f69393\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") "
Jul 27 17:47:40.992921 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.992707    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-tmp-dir\") pod \"e21648df-e6e0-4a36-9e2d-986d42f69393\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") "
Jul 27 17:47:40.992921 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.992734    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-kserve-provision-location\") pod \"e21648df-e6e0-4a36-9e2d-986d42f69393\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") "
Jul 27 17:47:40.992921 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.992779    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-model-cache\") pod \"e21648df-e6e0-4a36-9e2d-986d42f69393\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") "
Jul 27 17:47:40.992921 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.992808    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-home\") pod \"e21648df-e6e0-4a36-9e2d-986d42f69393\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") "
Jul 27 17:47:40.992921 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.992851    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-dshm\") pod \"e21648df-e6e0-4a36-9e2d-986d42f69393\" (UID: \"e21648df-e6e0-4a36-9e2d-986d42f69393\") "
Jul 27 17:47:40.993168 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.993087    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-model-cache" (OuterVolumeSpecName: "model-cache") pod "e21648df-e6e0-4a36-9e2d-986d42f69393" (UID: "e21648df-e6e0-4a36-9e2d-986d42f69393"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:40.993486 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.993459    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-home" (OuterVolumeSpecName: "home") pod "e21648df-e6e0-4a36-9e2d-986d42f69393" (UID: "e21648df-e6e0-4a36-9e2d-986d42f69393"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:40.995100 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.995070    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e21648df-e6e0-4a36-9e2d-986d42f69393-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e21648df-e6e0-4a36-9e2d-986d42f69393" (UID: "e21648df-e6e0-4a36-9e2d-986d42f69393"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:47:40.995221 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.995180    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21648df-e6e0-4a36-9e2d-986d42f69393-kube-api-access-p9d99" (OuterVolumeSpecName: "kube-api-access-p9d99") pod "e21648df-e6e0-4a36-9e2d-986d42f69393" (UID: "e21648df-e6e0-4a36-9e2d-986d42f69393"). InnerVolumeSpecName "kube-api-access-p9d99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:47:40.995394 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:40.995369    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-dshm" (OuterVolumeSpecName: "dshm") pod "e21648df-e6e0-4a36-9e2d-986d42f69393" (UID: "e21648df-e6e0-4a36-9e2d-986d42f69393"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:41.005920 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.005897    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e21648df-e6e0-4a36-9e2d-986d42f69393" (UID: "e21648df-e6e0-4a36-9e2d-986d42f69393"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:41.049733 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.049695    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e21648df-e6e0-4a36-9e2d-986d42f69393" (UID: "e21648df-e6e0-4a36-9e2d-986d42f69393"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:41.094076 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.094044    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:41.094076 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.094071    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:41.094247 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.094083    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:41.094247 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.094098    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9d99\" (UniqueName: \"kubernetes.io/projected/e21648df-e6e0-4a36-9e2d-986d42f69393-kube-api-access-p9d99\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:41.094247 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.094110    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21648df-e6e0-4a36-9e2d-986d42f69393-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:41.094247 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.094123    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:41.094247 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.094135    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21648df-e6e0-4a36-9e2d-986d42f69393-kserve-provision-location\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:47:41.207010 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.206978    2571 generic.go:356] "Generic (PLEG): container finished" podID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerID="bddb60878ba1432a42cdecdd38e6d018996956d2dc928898be092a3c21cfc722" exitCode=0
Jul 27 17:47:41.207171 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.207027    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"
Jul 27 17:47:41.207171 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.207051    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" event={"ID":"e21648df-e6e0-4a36-9e2d-986d42f69393","Type":"ContainerDied","Data":"bddb60878ba1432a42cdecdd38e6d018996956d2dc928898be092a3c21cfc722"}
Jul 27 17:47:41.207171 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.207101    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp" event={"ID":"e21648df-e6e0-4a36-9e2d-986d42f69393","Type":"ContainerDied","Data":"2b56277cb68bb910b33473beafc1bb85b5fe6989fd5c9df8f9b23a29e4dfb056"}
Jul 27 17:47:41.207171 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.207126    2571 scope.go:117] "RemoveContainer" containerID="bddb60878ba1432a42cdecdd38e6d018996956d2dc928898be092a3c21cfc722"
Jul 27 17:47:41.217896 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.217871    2571 scope.go:117] "RemoveContainer" containerID="4e6e64236ed82201ccdaaba7a58dd66508fda18e561e843e042505f57e4eb86e"
Jul 27 17:47:41.226061 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.226040    2571 scope.go:117] "RemoveContainer" containerID="bddb60878ba1432a42cdecdd38e6d018996956d2dc928898be092a3c21cfc722"
Jul 27 17:47:41.226298 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:47:41.226279    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bddb60878ba1432a42cdecdd38e6d018996956d2dc928898be092a3c21cfc722\": container with ID starting with bddb60878ba1432a42cdecdd38e6d018996956d2dc928898be092a3c21cfc722 not found: ID does not exist" containerID="bddb60878ba1432a42cdecdd38e6d018996956d2dc928898be092a3c21cfc722"
Jul 27 17:47:41.226363 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.226310    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bddb60878ba1432a42cdecdd38e6d018996956d2dc928898be092a3c21cfc722"} err="failed to get container status \"bddb60878ba1432a42cdecdd38e6d018996956d2dc928898be092a3c21cfc722\": rpc error: code = NotFound desc = could not find container \"bddb60878ba1432a42cdecdd38e6d018996956d2dc928898be092a3c21cfc722\": container with ID starting with bddb60878ba1432a42cdecdd38e6d018996956d2dc928898be092a3c21cfc722 not found: ID does not exist"
Jul 27 17:47:41.226363 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.226336    2571 scope.go:117] "RemoveContainer" containerID="4e6e64236ed82201ccdaaba7a58dd66508fda18e561e843e042505f57e4eb86e"
Jul 27 17:47:41.226621 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:47:41.226602    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6e64236ed82201ccdaaba7a58dd66508fda18e561e843e042505f57e4eb86e\": container with ID starting with 4e6e64236ed82201ccdaaba7a58dd66508fda18e561e843e042505f57e4eb86e not found: ID does not exist" containerID="4e6e64236ed82201ccdaaba7a58dd66508fda18e561e843e042505f57e4eb86e"
Jul 27 17:47:41.226671 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.226630    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6e64236ed82201ccdaaba7a58dd66508fda18e561e843e042505f57e4eb86e"} err="failed to get container status \"4e6e64236ed82201ccdaaba7a58dd66508fda18e561e843e042505f57e4eb86e\": rpc error: code = NotFound desc = could not find container \"4e6e64236ed82201ccdaaba7a58dd66508fda18e561e843e042505f57e4eb86e\": container with ID starting with 4e6e64236ed82201ccdaaba7a58dd66508fda18e561e843e042505f57e4eb86e not found: ID does not exist"
Jul 27 17:47:41.230578 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.230557    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"]
Jul 27 17:47:41.234064 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:41.234046    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fq9dgp"]
Jul 27 17:47:42.398444 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:47:42.398410    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" path="/var/lib/kubelet/pods/e21648df-e6e0-4a36-9e2d-986d42f69393/volumes"
Jul 27 17:48:03.736892 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.736806    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 27 17:48:03.737280 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.737147    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main"
Jul 27 17:48:03.737280 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.737160    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main"
Jul 27 17:48:03.737280 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.737175    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c740fca8-20c7-4d1c-ab5e-966a0d057146" containerName="istio-proxy"
Jul 27 17:48:03.737280 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.737181    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c740fca8-20c7-4d1c-ab5e-966a0d057146" containerName="istio-proxy"
Jul 27 17:48:03.737280 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.737191    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="storage-initializer"
Jul 27 17:48:03.737280 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.737198    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="storage-initializer"
Jul 27 17:48:03.737280 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.737269    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e21648df-e6e0-4a36-9e2d-986d42f69393" containerName="main"
Jul 27 17:48:03.737280 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.737275    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c740fca8-20c7-4d1c-ab5e-966a0d057146" containerName="istio-proxy"
Jul 27 17:48:03.740319 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.740300    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.744532 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.744475    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 27 17:48:03.744711 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.744635    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-bss89\""
Jul 27 17:48:03.744953 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.744929    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 27 17:48:03.745386 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.745363    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 27 17:48:03.745511 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.745421    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-f8v96\""
Jul 27 17:48:03.745511 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.745429    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 27 17:48:03.766599 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.766573    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 27 17:48:03.788219 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.788183    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.788349 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.788230    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.788349 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.788260    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.788349 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.788304    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.788349 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.788339    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qcf7\" (UniqueName: \"kubernetes.io/projected/e44e3ae0-58b5-4521-894e-90b3173a4774-kube-api-access-5qcf7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.788490 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.788399    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44e3ae0-58b5-4521-894e-90b3173a4774-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.788490 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.788430    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.870940 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.870906    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"]
Jul 27 17:48:03.874224 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.874207    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"
Jul 27 17:48:03.876956 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.876928    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-g6tln\""
Jul 27 17:48:03.885933 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.885909    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"]
Jul 27 17:48:03.889162 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.889142    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.889257 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.889172    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.889257 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.889194    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.889367 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.889316    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qcf7\" (UniqueName: \"kubernetes.io/projected/e44e3ae0-58b5-4521-894e-90b3173a4774-kube-api-access-5qcf7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.889421 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.889378    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76303864-2713-40ef-b802-62e4279cc65c-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z\" (UID: \"76303864-2713-40ef-b802-62e4279cc65c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"
Jul 27 17:48:03.889477 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.889436    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4fk\" (UniqueName: \"kubernetes.io/projected/76303864-2713-40ef-b802-62e4279cc65c-kube-api-access-tz4fk\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z\" (UID: \"76303864-2713-40ef-b802-62e4279cc65c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"
Jul 27 17:48:03.889559 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.889486    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.889559 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.889522    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.889559 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.889522    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44e3ae0-58b5-4521-894e-90b3173a4774-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.889712 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.889581    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.889712 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.889626    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.889869 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.889848    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.889915 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.889877    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.891350 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.891325    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.891843 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.891816    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44e3ae0-58b5-4521-894e-90b3173a4774-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.904231 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.904205    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qcf7\" (UniqueName: \"kubernetes.io/projected/e44e3ae0-58b5-4521-894e-90b3173a4774-kube-api-access-5qcf7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:03.990439 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.990348    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4fk\" (UniqueName: \"kubernetes.io/projected/76303864-2713-40ef-b802-62e4279cc65c-kube-api-access-tz4fk\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z\" (UID: \"76303864-2713-40ef-b802-62e4279cc65c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"
Jul 27 17:48:03.990621 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.990443    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76303864-2713-40ef-b802-62e4279cc65c-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z\" (UID: \"76303864-2713-40ef-b802-62e4279cc65c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"
Jul 27 17:48:03.992882 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:03.992853    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76303864-2713-40ef-b802-62e4279cc65c-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z\" (UID: \"76303864-2713-40ef-b802-62e4279cc65c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"
Jul 27 17:48:04.003686 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:04.003661    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4fk\" (UniqueName: \"kubernetes.io/projected/76303864-2713-40ef-b802-62e4279cc65c-kube-api-access-tz4fk\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z\" (UID: \"76303864-2713-40ef-b802-62e4279cc65c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"
Jul 27 17:48:04.051623 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:04.051591    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:04.184712 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:04.184675    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"
Jul 27 17:48:04.187656 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:04.187628    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 27 17:48:04.193270 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:48:04.193243    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44e3ae0_58b5_4521_894e_90b3173a4774.slice/crio-f95d73e16a0a0fc7f1b461a6248f847dcffea5c680f96839df378ba7fb78e37b WatchSource:0}: Error finding container f95d73e16a0a0fc7f1b461a6248f847dcffea5c680f96839df378ba7fb78e37b: Status 404 returned error can't find the container with id f95d73e16a0a0fc7f1b461a6248f847dcffea5c680f96839df378ba7fb78e37b
Jul 27 17:48:04.327375 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:04.327181    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e44e3ae0-58b5-4521-894e-90b3173a4774","Type":"ContainerStarted","Data":"7c40cb93a313b551adc54955aafd2a90fd01c42d0b45dc1946dc943054daa4c6"}
Jul 27 17:48:04.327375 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:04.327237    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e44e3ae0-58b5-4521-894e-90b3173a4774","Type":"ContainerStarted","Data":"f95d73e16a0a0fc7f1b461a6248f847dcffea5c680f96839df378ba7fb78e37b"}
Jul 27 17:48:04.327911 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:04.327890    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"]
Jul 27 17:48:04.329632 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:48:04.329608    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76303864_2713_40ef_b802_62e4279cc65c.slice/crio-f3866983a0bd8f39ee280e9e1629dd1a51d75241d1ce979d851ba7bf087e172b WatchSource:0}: Error finding container f3866983a0bd8f39ee280e9e1629dd1a51d75241d1ce979d851ba7bf087e172b: Status 404 returned error can't find the container with id f3866983a0bd8f39ee280e9e1629dd1a51d75241d1ce979d851ba7bf087e172b
Jul 27 17:48:04.332255 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:04.332238    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:48:05.338512 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:05.338442    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z" event={"ID":"76303864-2713-40ef-b802-62e4279cc65c","Type":"ContainerStarted","Data":"f3866983a0bd8f39ee280e9e1629dd1a51d75241d1ce979d851ba7bf087e172b"}
Jul 27 17:48:07.350234 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:07.350196    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z" event={"ID":"76303864-2713-40ef-b802-62e4279cc65c","Type":"ContainerStarted","Data":"9694af63cf65b3f033b17055046b8109fb3876bfbe8274dad071ae0c14254451"}
Jul 27 17:48:07.350735 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:07.350285    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"
Jul 27 17:48:07.371481 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:07.371421    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z" podStartSLOduration=1.92562441 podStartE2EDuration="4.371398717s" totalImagesPullingTime="2.445774307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:48:07.369714019 +0000 UTC m=+1169.414452098" watchObservedRunningTime="2026-07-27 17:48:07.371398717 +0000 UTC m=+1169.416136800"
Jul 27 17:48:12.379342 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:12.379308    2571 generic.go:356] "Generic (PLEG): container finished" podID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerID="7c40cb93a313b551adc54955aafd2a90fd01c42d0b45dc1946dc943054daa4c6" exitCode=0
Jul 27 17:48:12.379759 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:12.379381    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e44e3ae0-58b5-4521-894e-90b3173a4774","Type":"ContainerDied","Data":"7c40cb93a313b551adc54955aafd2a90fd01c42d0b45dc1946dc943054daa4c6"}
Jul 27 17:48:13.387059 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:13.387028    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e44e3ae0-58b5-4521-894e-90b3173a4774","Type":"ContainerStarted","Data":"830f5b0a9198b97cfdd20e6a2fdb73c95f8d54d474a80844bac84e2f5c7bed1b"}
Jul 27 17:48:13.414870 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:13.414806    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.256477787 podStartE2EDuration="10.414782505s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.158304718s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:48:13.411420253 +0000 UTC m=+1175.456158334" watchObservedRunningTime="2026-07-27 17:48:13.414782505 +0000 UTC m=+1175.459520588"
Jul 27 17:48:14.052212 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:14.052166    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:14.052392 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:14.052246    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:48:14.053650 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:14.053623    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 27 17:48:24.052749 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:24.052705    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 27 17:48:34.052939 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:34.052892    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 27 17:48:38.357489 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:38.357459    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"
Jul 27 17:48:44.052552 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:44.052474    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 27 17:48:54.052539 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:48:54.052476    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 27 17:49:04.052464 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:49:04.052421    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 27 17:49:14.052840 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:49:14.052782    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 27 17:49:24.052350 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:49:24.052306    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 27 17:49:34.053099 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:49:34.052999    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 27 17:49:44.061759 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:49:44.061726    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:49:44.069238 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:49:44.069215    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:50:53.295180 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:50:53.295142    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"]
Jul 27 17:50:53.303262 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:50:53.303234    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 27 17:50:58.357369 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:50:58.357332    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z" podUID="76303864-2713-40ef-b802-62e4279cc65c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:51:08.314373 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:08.314276    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z" podUID="76303864-2713-40ef-b802-62e4279cc65c" containerName="main" containerID="cri-o://9694af63cf65b3f033b17055046b8109fb3876bfbe8274dad071ae0c14254451" gracePeriod=45
Jul 27 17:51:08.324963 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:08.324910    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main" containerID="cri-o://830f5b0a9198b97cfdd20e6a2fdb73c95f8d54d474a80844bac84e2f5c7bed1b" gracePeriod=45
Jul 27 17:51:08.356788 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:51:08.356760    2571 logging.go:55] [core] [Channel #110 SubChannel #111]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.37:9003", ServerName: "10.133.0.37:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.37:9003: connect: connection refused"
Jul 27 17:51:08.564729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:08.564656    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"
Jul 27 17:51:08.612577 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:08.612544    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76303864-2713-40ef-b802-62e4279cc65c-tls-certs\") pod \"76303864-2713-40ef-b802-62e4279cc65c\" (UID: \"76303864-2713-40ef-b802-62e4279cc65c\") "
Jul 27 17:51:08.612758 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:08.612652    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz4fk\" (UniqueName: \"kubernetes.io/projected/76303864-2713-40ef-b802-62e4279cc65c-kube-api-access-tz4fk\") pod \"76303864-2713-40ef-b802-62e4279cc65c\" (UID: \"76303864-2713-40ef-b802-62e4279cc65c\") "
Jul 27 17:51:08.614802 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:08.614767    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76303864-2713-40ef-b802-62e4279cc65c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76303864-2713-40ef-b802-62e4279cc65c" (UID: "76303864-2713-40ef-b802-62e4279cc65c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:51:08.614911 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:08.614837    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76303864-2713-40ef-b802-62e4279cc65c-kube-api-access-tz4fk" (OuterVolumeSpecName: "kube-api-access-tz4fk") pod "76303864-2713-40ef-b802-62e4279cc65c" (UID: "76303864-2713-40ef-b802-62e4279cc65c"). InnerVolumeSpecName "kube-api-access-tz4fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:51:08.713680 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:08.713645    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz4fk\" (UniqueName: \"kubernetes.io/projected/76303864-2713-40ef-b802-62e4279cc65c-kube-api-access-tz4fk\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:51:08.713680 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:08.713677    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76303864-2713-40ef-b802-62e4279cc65c-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:51:09.072925 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.072901    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:51:09.116936 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.116859    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-home\") pod \"e44e3ae0-58b5-4521-894e-90b3173a4774\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") "
Jul 27 17:51:09.116936 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.116907    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-dshm\") pod \"e44e3ae0-58b5-4521-894e-90b3173a4774\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") "
Jul 27 17:51:09.117156 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.116943    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-model-cache\") pod \"e44e3ae0-58b5-4521-894e-90b3173a4774\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") "
Jul 27 17:51:09.117156 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.116961    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qcf7\" (UniqueName: \"kubernetes.io/projected/e44e3ae0-58b5-4521-894e-90b3173a4774-kube-api-access-5qcf7\") pod \"e44e3ae0-58b5-4521-894e-90b3173a4774\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") "
Jul 27 17:51:09.117156 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.116985    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44e3ae0-58b5-4521-894e-90b3173a4774-tls-certs\") pod \"e44e3ae0-58b5-4521-894e-90b3173a4774\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") "
Jul 27 17:51:09.117156 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.117002    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-kserve-provision-location\") pod \"e44e3ae0-58b5-4521-894e-90b3173a4774\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") "
Jul 27 17:51:09.117156 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.117021    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-tmp-dir\") pod \"e44e3ae0-58b5-4521-894e-90b3173a4774\" (UID: \"e44e3ae0-58b5-4521-894e-90b3173a4774\") "
Jul 27 17:51:09.117384 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.117221    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-model-cache" (OuterVolumeSpecName: "model-cache") pod "e44e3ae0-58b5-4521-894e-90b3173a4774" (UID: "e44e3ae0-58b5-4521-894e-90b3173a4774"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:09.117609 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.117584    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-home" (OuterVolumeSpecName: "home") pod "e44e3ae0-58b5-4521-894e-90b3173a4774" (UID: "e44e3ae0-58b5-4521-894e-90b3173a4774"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:09.119085 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.119062    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e44e3ae0-58b5-4521-894e-90b3173a4774-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e44e3ae0-58b5-4521-894e-90b3173a4774" (UID: "e44e3ae0-58b5-4521-894e-90b3173a4774"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:51:09.119194 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.119159    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44e3ae0-58b5-4521-894e-90b3173a4774-kube-api-access-5qcf7" (OuterVolumeSpecName: "kube-api-access-5qcf7") pod "e44e3ae0-58b5-4521-894e-90b3173a4774" (UID: "e44e3ae0-58b5-4521-894e-90b3173a4774"). InnerVolumeSpecName "kube-api-access-5qcf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:51:09.119268 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.119255    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-dshm" (OuterVolumeSpecName: "dshm") pod "e44e3ae0-58b5-4521-894e-90b3173a4774" (UID: "e44e3ae0-58b5-4521-894e-90b3173a4774"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:09.129555 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.129527    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e44e3ae0-58b5-4521-894e-90b3173a4774" (UID: "e44e3ae0-58b5-4521-894e-90b3173a4774"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:09.177719 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.177679    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e44e3ae0-58b5-4521-894e-90b3173a4774" (UID: "e44e3ae0-58b5-4521-894e-90b3173a4774"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:09.218172 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.218137    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:51:09.218172 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.218168    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qcf7\" (UniqueName: \"kubernetes.io/projected/e44e3ae0-58b5-4521-894e-90b3173a4774-kube-api-access-5qcf7\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:51:09.218172 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.218179    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44e3ae0-58b5-4521-894e-90b3173a4774-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:51:09.218397 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.218189    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-kserve-provision-location\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:51:09.218397 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.218219    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:51:09.218397 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.218228    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:51:09.218397 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.218235    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44e3ae0-58b5-4521-894e-90b3173a4774-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:51:09.281626 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.281592    2571 generic.go:356] "Generic (PLEG): container finished" podID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerID="830f5b0a9198b97cfdd20e6a2fdb73c95f8d54d474a80844bac84e2f5c7bed1b" exitCode=0
Jul 27 17:51:09.281796 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.281644    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:51:09.281796 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.281678    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e44e3ae0-58b5-4521-894e-90b3173a4774","Type":"ContainerDied","Data":"830f5b0a9198b97cfdd20e6a2fdb73c95f8d54d474a80844bac84e2f5c7bed1b"}
Jul 27 17:51:09.281796 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.281767    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e44e3ae0-58b5-4521-894e-90b3173a4774","Type":"ContainerDied","Data":"f95d73e16a0a0fc7f1b461a6248f847dcffea5c680f96839df378ba7fb78e37b"}
Jul 27 17:51:09.281948 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.281798    2571 scope.go:117] "RemoveContainer" containerID="830f5b0a9198b97cfdd20e6a2fdb73c95f8d54d474a80844bac84e2f5c7bed1b"
Jul 27 17:51:09.283187 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.283162    2571 generic.go:356] "Generic (PLEG): container finished" podID="76303864-2713-40ef-b802-62e4279cc65c" containerID="9694af63cf65b3f033b17055046b8109fb3876bfbe8274dad071ae0c14254451" exitCode=0
Jul 27 17:51:09.283299 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.283223    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"
Jul 27 17:51:09.283363 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.283221    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z" event={"ID":"76303864-2713-40ef-b802-62e4279cc65c","Type":"ContainerDied","Data":"9694af63cf65b3f033b17055046b8109fb3876bfbe8274dad071ae0c14254451"}
Jul 27 17:51:09.283363 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.283336    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z" event={"ID":"76303864-2713-40ef-b802-62e4279cc65c","Type":"ContainerDied","Data":"f3866983a0bd8f39ee280e9e1629dd1a51d75241d1ce979d851ba7bf087e172b"}
Jul 27 17:51:09.291665 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.291648    2571 scope.go:117] "RemoveContainer" containerID="7c40cb93a313b551adc54955aafd2a90fd01c42d0b45dc1946dc943054daa4c6"
Jul 27 17:51:09.299822 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.299801    2571 scope.go:117] "RemoveContainer" containerID="830f5b0a9198b97cfdd20e6a2fdb73c95f8d54d474a80844bac84e2f5c7bed1b"
Jul 27 17:51:09.300047 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:51:09.300029    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830f5b0a9198b97cfdd20e6a2fdb73c95f8d54d474a80844bac84e2f5c7bed1b\": container with ID starting with 830f5b0a9198b97cfdd20e6a2fdb73c95f8d54d474a80844bac84e2f5c7bed1b not found: ID does not exist" containerID="830f5b0a9198b97cfdd20e6a2fdb73c95f8d54d474a80844bac84e2f5c7bed1b"
Jul 27 17:51:09.300099 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.300055    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830f5b0a9198b97cfdd20e6a2fdb73c95f8d54d474a80844bac84e2f5c7bed1b"} err="failed to get container status \"830f5b0a9198b97cfdd20e6a2fdb73c95f8d54d474a80844bac84e2f5c7bed1b\": rpc error: code = NotFound desc = could not find container \"830f5b0a9198b97cfdd20e6a2fdb73c95f8d54d474a80844bac84e2f5c7bed1b\": container with ID starting with 830f5b0a9198b97cfdd20e6a2fdb73c95f8d54d474a80844bac84e2f5c7bed1b not found: ID does not exist"
Jul 27 17:51:09.300099 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.300075    2571 scope.go:117] "RemoveContainer" containerID="7c40cb93a313b551adc54955aafd2a90fd01c42d0b45dc1946dc943054daa4c6"
Jul 27 17:51:09.300260 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:51:09.300243    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c40cb93a313b551adc54955aafd2a90fd01c42d0b45dc1946dc943054daa4c6\": container with ID starting with 7c40cb93a313b551adc54955aafd2a90fd01c42d0b45dc1946dc943054daa4c6 not found: ID does not exist" containerID="7c40cb93a313b551adc54955aafd2a90fd01c42d0b45dc1946dc943054daa4c6"
Jul 27 17:51:09.300296 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.300267    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c40cb93a313b551adc54955aafd2a90fd01c42d0b45dc1946dc943054daa4c6"} err="failed to get container status \"7c40cb93a313b551adc54955aafd2a90fd01c42d0b45dc1946dc943054daa4c6\": rpc error: code = NotFound desc = could not find container \"7c40cb93a313b551adc54955aafd2a90fd01c42d0b45dc1946dc943054daa4c6\": container with ID starting with 7c40cb93a313b551adc54955aafd2a90fd01c42d0b45dc1946dc943054daa4c6 not found: ID does not exist"
Jul 27 17:51:09.300296 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.300283    2571 scope.go:117] "RemoveContainer" containerID="9694af63cf65b3f033b17055046b8109fb3876bfbe8274dad071ae0c14254451"
Jul 27 17:51:09.306113 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.306093    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 27 17:51:09.308019 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.308000    2571 scope.go:117] "RemoveContainer" containerID="9694af63cf65b3f033b17055046b8109fb3876bfbe8274dad071ae0c14254451"
Jul 27 17:51:09.308272 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:51:09.308253    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9694af63cf65b3f033b17055046b8109fb3876bfbe8274dad071ae0c14254451\": container with ID starting with 9694af63cf65b3f033b17055046b8109fb3876bfbe8274dad071ae0c14254451 not found: ID does not exist" containerID="9694af63cf65b3f033b17055046b8109fb3876bfbe8274dad071ae0c14254451"
Jul 27 17:51:09.308322 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.308281    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9694af63cf65b3f033b17055046b8109fb3876bfbe8274dad071ae0c14254451"} err="failed to get container status \"9694af63cf65b3f033b17055046b8109fb3876bfbe8274dad071ae0c14254451\": rpc error: code = NotFound desc = could not find container \"9694af63cf65b3f033b17055046b8109fb3876bfbe8274dad071ae0c14254451\": container with ID starting with 9694af63cf65b3f033b17055046b8109fb3876bfbe8274dad071ae0c14254451 not found: ID does not exist"
Jul 27 17:51:09.312221 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.312202    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 27 17:51:09.323644 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.323618    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"]
Jul 27 17:51:09.329083 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.329063    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z"]
Jul 27 17:51:09.357090 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:09.357056    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schentk6z" podUID="76303864-2713-40ef-b802-62e4279cc65c" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.37:9003\" within 1s: context deadline exceeded"
Jul 27 17:51:10.063604 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:10.063557    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 27 17:51:10.399049 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:10.398968    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76303864-2713-40ef-b802-62e4279cc65c" path="/var/lib/kubelet/pods/76303864-2713-40ef-b802-62e4279cc65c/volumes"
Jul 27 17:51:10.399405 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:10.399337    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" path="/var/lib/kubelet/pods/e44e3ae0-58b5-4521-894e-90b3173a4774/volumes"
Jul 27 17:51:14.119855 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.119819    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"]
Jul 27 17:51:14.120352 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.120335    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main"
Jul 27 17:51:14.120429 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.120353    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main"
Jul 27 17:51:14.120429 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.120381    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="storage-initializer"
Jul 27 17:51:14.120429 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.120392    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="storage-initializer"
Jul 27 17:51:14.120429 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.120405    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76303864-2713-40ef-b802-62e4279cc65c" containerName="main"
Jul 27 17:51:14.120429 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.120413    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="76303864-2713-40ef-b802-62e4279cc65c" containerName="main"
Jul 27 17:51:14.120696 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.120533    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="76303864-2713-40ef-b802-62e4279cc65c" containerName="main"
Jul 27 17:51:14.120696 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.120546    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e44e3ae0-58b5-4521-894e-90b3173a4774" containerName="main"
Jul 27 17:51:14.125274 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.125253    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.128179 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.128156    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 27 17:51:14.128712 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.128697    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Jul 27 17:51:14.128774 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.128698    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 27 17:51:14.128774 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.128699    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-xh999\""
Jul 27 17:51:14.129668 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.129647    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 27 17:51:14.135805 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.135784    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"]
Jul 27 17:51:14.154409 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.154383    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bwt9\" (UniqueName: \"kubernetes.io/projected/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-kube-api-access-6bwt9\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.154535 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.154427    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-model-cache\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.154535 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.154461    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.154535 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.154530    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-home\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.154700 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.154565    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-tmp-dir\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.154700 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.154589    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-tls-certs\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.154700 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.154651    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-dshm\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.255640 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.255605    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bwt9\" (UniqueName: \"kubernetes.io/projected/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-kube-api-access-6bwt9\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.255640 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.255644    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-model-cache\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.255881 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.255666    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.255881 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.255702    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-home\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.255881 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.255732    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-tmp-dir\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.255881 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.255753    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-tls-certs\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.256109 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.255881    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-dshm\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.256164 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.256112    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.256164 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.256130    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-tmp-dir\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.256231 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.256164    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-model-cache\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.256231 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.256191    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-home\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.258110 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.258080    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-dshm\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.258378 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.258360    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-tls-certs\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.268046 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.268017    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bwt9\" (UniqueName: \"kubernetes.io/projected/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-kube-api-access-6bwt9\") pod \"auth-invalid-token-test-kserve-5ff8967c79-s6sn6\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.435598 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.435492    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:14.583824 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:14.582778    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"]
Jul 27 17:51:14.585836 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:51:14.585804    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd06ee3_6a5e_4466_b38b_01c4b872f86a.slice/crio-deea67bcfba6c54ae79cae7f1459b38e0ba2c286a5f6c66ae06da885af6d2e9d WatchSource:0}: Error finding container deea67bcfba6c54ae79cae7f1459b38e0ba2c286a5f6c66ae06da885af6d2e9d: Status 404 returned error can't find the container with id deea67bcfba6c54ae79cae7f1459b38e0ba2c286a5f6c66ae06da885af6d2e9d
Jul 27 17:51:15.316529 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:15.316483    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" event={"ID":"dfd06ee3-6a5e-4466-b38b-01c4b872f86a","Type":"ContainerStarted","Data":"97648a8701a705bf756371cde08c96966110462a0e8b9573797f63ced5161894"}
Jul 27 17:51:15.316908 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:15.316547    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" event={"ID":"dfd06ee3-6a5e-4466-b38b-01c4b872f86a","Type":"ContainerStarted","Data":"deea67bcfba6c54ae79cae7f1459b38e0ba2c286a5f6c66ae06da885af6d2e9d"}
Jul 27 17:51:21.350625 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:21.350594    2571 generic.go:356] "Generic (PLEG): container finished" podID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerID="97648a8701a705bf756371cde08c96966110462a0e8b9573797f63ced5161894" exitCode=0
Jul 27 17:51:21.350958 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:21.350667    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" event={"ID":"dfd06ee3-6a5e-4466-b38b-01c4b872f86a","Type":"ContainerDied","Data":"97648a8701a705bf756371cde08c96966110462a0e8b9573797f63ced5161894"}
Jul 27 17:51:22.358471 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:22.358437    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" event={"ID":"dfd06ee3-6a5e-4466-b38b-01c4b872f86a","Type":"ContainerStarted","Data":"25f58ce1082ef84c30a72c2db2e4518fece02600e9dad9621137ea6eb58974a9"}
Jul 27 17:51:22.386449 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:22.386395    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podStartSLOduration=1.721646145 podStartE2EDuration="8.386377096s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.664730951s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:51:22.383175309 +0000 UTC m=+1364.427913391" watchObservedRunningTime="2026-07-27 17:51:22.386377096 +0000 UTC m=+1364.431115178"
Jul 27 17:51:22.436489 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:22.436459    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:24.435951 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:24.435913    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:51:24.437572 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:24.437545    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 27 17:51:34.436464 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:34.436414    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 27 17:51:44.436657 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:44.436614    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 27 17:51:54.436868 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:51:54.436824    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 27 17:52:04.436355 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:04.436304    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 27 17:52:14.436120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:14.436071    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 27 17:52:21.042066 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.042032    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"]
Jul 27 17:52:21.045674 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.045650    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"
Jul 27 17:52:21.049761 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.049737    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-qxwwh\""
Jul 27 17:52:21.049885 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.049749    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-tl45v\""
Jul 27 17:52:21.049885 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.049844    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 27 17:52:21.049997 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.049894    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 27 17:52:21.050090 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.050071    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 27 17:52:21.056774 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.056749    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"]
Jul 27 17:52:21.149847 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.149815    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkbb6\" (UniqueName: \"kubernetes.io/projected/6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3-kube-api-access-zkbb6\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g\" (UID: \"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"
Jul 27 17:52:21.150027 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.149871    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g\" (UID: \"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"
Jul 27 17:52:21.250407 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.250367    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g\" (UID: \"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"
Jul 27 17:52:21.250612 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.250444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkbb6\" (UniqueName: \"kubernetes.io/projected/6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3-kube-api-access-zkbb6\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g\" (UID: \"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"
Jul 27 17:52:21.253284 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.253253    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g\" (UID: \"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"
Jul 27 17:52:21.258828 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.258798    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkbb6\" (UniqueName: \"kubernetes.io/projected/6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3-kube-api-access-zkbb6\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g\" (UID: \"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"
Jul 27 17:52:21.359152 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.359066    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"
Jul 27 17:52:21.489590 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.489552    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"]
Jul 27 17:52:21.489915 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:52:21.489887    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed82b6a_7d75_4e4d_be4d_27e7eb4f86c3.slice/crio-3525adca24155b72d8145332ba092389a23f0a0ffa632a2c5f2dee9c3564715a WatchSource:0}: Error finding container 3525adca24155b72d8145332ba092389a23f0a0ffa632a2c5f2dee9c3564715a: Status 404 returned error can't find the container with id 3525adca24155b72d8145332ba092389a23f0a0ffa632a2c5f2dee9c3564715a
Jul 27 17:52:21.671607 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.671515    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g" event={"ID":"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3","Type":"ContainerStarted","Data":"5e256bb6b96bc9e3ce762719327900944aad8206d28ea2e4358fdeb36728d41f"}
Jul 27 17:52:21.671607 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.671573    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g" event={"ID":"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3","Type":"ContainerStarted","Data":"3525adca24155b72d8145332ba092389a23f0a0ffa632a2c5f2dee9c3564715a"}
Jul 27 17:52:21.671802 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.671649    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"
Jul 27 17:52:21.690516 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:21.690453    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g" podStartSLOduration=0.690438763 podStartE2EDuration="690.438763ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:52:21.689479789 +0000 UTC m=+1423.734217958" watchObservedRunningTime="2026-07-27 17:52:21.690438763 +0000 UTC m=+1423.735176843"
Jul 27 17:52:24.436700 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:24.436662    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 27 17:52:34.436899 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:34.436855    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 27 17:52:44.436120 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:44.436074    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 27 17:52:52.679517 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:52.679475    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"
Jul 27 17:52:54.104299 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:54.104266    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"]
Jul 27 17:52:54.436393 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:52:54.436284    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 27 17:53:02.679289 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:02.679249    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g" podUID="6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:53:04.436263 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:04.436218    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 27 17:53:09.124040 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.123957    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g" podUID="6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3" containerName="main" containerID="cri-o://5e256bb6b96bc9e3ce762719327900944aad8206d28ea2e4358fdeb36728d41f" gracePeriod=45
Jul 27 17:53:09.378965 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.378903    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"
Jul 27 17:53:09.479924 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.479889    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkbb6\" (UniqueName: \"kubernetes.io/projected/6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3-kube-api-access-zkbb6\") pod \"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3\" (UID: \"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3\") "
Jul 27 17:53:09.480103 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.479941    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3-tls-certs\") pod \"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3\" (UID: \"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3\") "
Jul 27 17:53:09.482127 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.482088    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3-kube-api-access-zkbb6" (OuterVolumeSpecName: "kube-api-access-zkbb6") pod "6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3" (UID: "6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3"). InnerVolumeSpecName "kube-api-access-zkbb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:53:09.482127 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.482102    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3" (UID: "6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:53:09.580810 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.580773    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkbb6\" (UniqueName: \"kubernetes.io/projected/6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3-kube-api-access-zkbb6\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:53:09.580810 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.580806    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:53:09.933769 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.932808    2571 generic.go:356] "Generic (PLEG): container finished" podID="6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3" containerID="5e256bb6b96bc9e3ce762719327900944aad8206d28ea2e4358fdeb36728d41f" exitCode=0
Jul 27 17:53:09.933769 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.932917    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g" event={"ID":"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3","Type":"ContainerDied","Data":"5e256bb6b96bc9e3ce762719327900944aad8206d28ea2e4358fdeb36728d41f"}
Jul 27 17:53:09.933769 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.932972    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g" event={"ID":"6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3","Type":"ContainerDied","Data":"3525adca24155b72d8145332ba092389a23f0a0ffa632a2c5f2dee9c3564715a"}
Jul 27 17:53:09.933769 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.933004    2571 scope.go:117] "RemoveContainer" containerID="5e256bb6b96bc9e3ce762719327900944aad8206d28ea2e4358fdeb36728d41f"
Jul 27 17:53:09.933769 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.933177    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"
Jul 27 17:53:09.946457 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.946436    2571 scope.go:117] "RemoveContainer" containerID="5e256bb6b96bc9e3ce762719327900944aad8206d28ea2e4358fdeb36728d41f"
Jul 27 17:53:09.946771 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:53:09.946752    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e256bb6b96bc9e3ce762719327900944aad8206d28ea2e4358fdeb36728d41f\": container with ID starting with 5e256bb6b96bc9e3ce762719327900944aad8206d28ea2e4358fdeb36728d41f not found: ID does not exist" containerID="5e256bb6b96bc9e3ce762719327900944aad8206d28ea2e4358fdeb36728d41f"
Jul 27 17:53:09.946832 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.946781    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e256bb6b96bc9e3ce762719327900944aad8206d28ea2e4358fdeb36728d41f"} err="failed to get container status \"5e256bb6b96bc9e3ce762719327900944aad8206d28ea2e4358fdeb36728d41f\": rpc error: code = NotFound desc = could not find container \"5e256bb6b96bc9e3ce762719327900944aad8206d28ea2e4358fdeb36728d41f\": container with ID starting with 5e256bb6b96bc9e3ce762719327900944aad8206d28ea2e4358fdeb36728d41f not found: ID does not exist"
Jul 27 17:53:09.959634 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.959608    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"]
Jul 27 17:53:09.963711 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:09.963686    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5qr87g"]
Jul 27 17:53:10.400018 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:10.399975    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3" path="/var/lib/kubelet/pods/6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3/volumes"
Jul 27 17:53:14.446023 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:14.445987    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:53:14.454376 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:14.454339    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:53:25.130045 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.130009    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"]
Jul 27 17:53:25.130627 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.130457    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3" containerName="main"
Jul 27 17:53:25.130627 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.130476    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3" containerName="main"
Jul 27 17:53:25.130627 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.130587    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed82b6a-7d75-4e4d-be4d-27e7eb4f86c3" containerName="main"
Jul 27 17:53:25.134707 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.134682    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"
Jul 27 17:53:25.138717 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.138690    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 27 17:53:25.138846 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.138696    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 27 17:53:25.138846 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.138762    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-trnx4\""
Jul 27 17:53:25.138846 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.138800    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-plltt\""
Jul 27 17:53:25.139057 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.139040    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 27 17:53:25.145319 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.145296    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"]
Jul 27 17:53:25.219551 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.219513    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc9fb26-e200-4452-83fe-43f15ba5e2da-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl\" (UID: \"7bc9fb26-e200-4452-83fe-43f15ba5e2da\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"
Jul 27 17:53:25.219723 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.219562    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wskwd\" (UniqueName: \"kubernetes.io/projected/7bc9fb26-e200-4452-83fe-43f15ba5e2da-kube-api-access-wskwd\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl\" (UID: \"7bc9fb26-e200-4452-83fe-43f15ba5e2da\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"
Jul 27 17:53:25.320641 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.320607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc9fb26-e200-4452-83fe-43f15ba5e2da-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl\" (UID: \"7bc9fb26-e200-4452-83fe-43f15ba5e2da\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"
Jul 27 17:53:25.320641 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.320645    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wskwd\" (UniqueName: \"kubernetes.io/projected/7bc9fb26-e200-4452-83fe-43f15ba5e2da-kube-api-access-wskwd\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl\" (UID: \"7bc9fb26-e200-4452-83fe-43f15ba5e2da\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"
Jul 27 17:53:25.323061 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.323037    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc9fb26-e200-4452-83fe-43f15ba5e2da-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl\" (UID: \"7bc9fb26-e200-4452-83fe-43f15ba5e2da\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"
Jul 27 17:53:25.329189 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.329163    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wskwd\" (UniqueName: \"kubernetes.io/projected/7bc9fb26-e200-4452-83fe-43f15ba5e2da-kube-api-access-wskwd\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl\" (UID: \"7bc9fb26-e200-4452-83fe-43f15ba5e2da\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"
Jul 27 17:53:25.446405 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.446298    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"
Jul 27 17:53:25.576911 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:25.576887    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"]
Jul 27 17:53:25.579595 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:53:25.579560    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc9fb26_e200_4452_83fe_43f15ba5e2da.slice/crio-f63b43c41dc23fe74fd6a6a3368e768869d5c1abe07bf83cc7ab3a4a58f070fc WatchSource:0}: Error finding container f63b43c41dc23fe74fd6a6a3368e768869d5c1abe07bf83cc7ab3a4a58f070fc: Status 404 returned error can't find the container with id f63b43c41dc23fe74fd6a6a3368e768869d5c1abe07bf83cc7ab3a4a58f070fc
Jul 27 17:53:26.014221 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:26.014181    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl" event={"ID":"7bc9fb26-e200-4452-83fe-43f15ba5e2da","Type":"ContainerStarted","Data":"e3121f0861f91ef2523ba270a8e2b6c8b9eb8a612bbd41c86aa694c53d21caca"}
Jul 27 17:53:26.014400 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:26.014250    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl" event={"ID":"7bc9fb26-e200-4452-83fe-43f15ba5e2da","Type":"ContainerStarted","Data":"f63b43c41dc23fe74fd6a6a3368e768869d5c1abe07bf83cc7ab3a4a58f070fc"}
Jul 27 17:53:26.014400 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:26.014299    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"
Jul 27 17:53:26.054583 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:26.054529    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl" podStartSLOduration=1.054512891 podStartE2EDuration="1.054512891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:53:26.052683058 +0000 UTC m=+1488.097421138" watchObservedRunningTime="2026-07-27 17:53:26.054512891 +0000 UTC m=+1488.099250965"
Jul 27 17:53:34.803228 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:34.803184    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"]
Jul 27 17:53:49.825041 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:49.824990    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main" containerID="cri-o://25f58ce1082ef84c30a72c2db2e4518fece02600e9dad9621137ea6eb58974a9" gracePeriod=45
Jul 27 17:53:50.484989 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.484964    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:53:50.637070 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.636980    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-home\") pod \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") "
Jul 27 17:53:50.637070 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.637034    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-kserve-provision-location\") pod \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") "
Jul 27 17:53:50.637070 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.637066    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bwt9\" (UniqueName: \"kubernetes.io/projected/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-kube-api-access-6bwt9\") pod \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") "
Jul 27 17:53:50.637518 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.637141    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-dshm\") pod \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") "
Jul 27 17:53:50.637518 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.637182    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-tls-certs\") pod \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") "
Jul 27 17:53:50.637518 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.637210    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-model-cache\") pod \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") "
Jul 27 17:53:50.637518 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.637354    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-tmp-dir\") pod \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\" (UID: \"dfd06ee3-6a5e-4466-b38b-01c4b872f86a\") "
Jul 27 17:53:50.637518 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.637436    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-model-cache" (OuterVolumeSpecName: "model-cache") pod "dfd06ee3-6a5e-4466-b38b-01c4b872f86a" (UID: "dfd06ee3-6a5e-4466-b38b-01c4b872f86a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:53:50.637862 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.637653    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:53:50.637862 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.637813    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-home" (OuterVolumeSpecName: "home") pod "dfd06ee3-6a5e-4466-b38b-01c4b872f86a" (UID: "dfd06ee3-6a5e-4466-b38b-01c4b872f86a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:53:50.639568 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.639529    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-kube-api-access-6bwt9" (OuterVolumeSpecName: "kube-api-access-6bwt9") pod "dfd06ee3-6a5e-4466-b38b-01c4b872f86a" (UID: "dfd06ee3-6a5e-4466-b38b-01c4b872f86a"). InnerVolumeSpecName "kube-api-access-6bwt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:53:50.639808 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.639785    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-dshm" (OuterVolumeSpecName: "dshm") pod "dfd06ee3-6a5e-4466-b38b-01c4b872f86a" (UID: "dfd06ee3-6a5e-4466-b38b-01c4b872f86a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:53:50.639888 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.639871    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dfd06ee3-6a5e-4466-b38b-01c4b872f86a" (UID: "dfd06ee3-6a5e-4466-b38b-01c4b872f86a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:53:50.656057 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.656031    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dfd06ee3-6a5e-4466-b38b-01c4b872f86a" (UID: "dfd06ee3-6a5e-4466-b38b-01c4b872f86a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:53:50.701448 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.701416    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dfd06ee3-6a5e-4466-b38b-01c4b872f86a" (UID: "dfd06ee3-6a5e-4466-b38b-01c4b872f86a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:53:50.738947 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.738919    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:53:50.738947 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.738944    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:53:50.738947 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.738953    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:53:50.739125 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.738961    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:53:50.739125 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.738969    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-kserve-provision-location\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:53:50.739125 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:50.738981    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bwt9\" (UniqueName: \"kubernetes.io/projected/dfd06ee3-6a5e-4466-b38b-01c4b872f86a-kube-api-access-6bwt9\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:53:51.144929 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:51.144895    2571 generic.go:356] "Generic (PLEG): container finished" podID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerID="25f58ce1082ef84c30a72c2db2e4518fece02600e9dad9621137ea6eb58974a9" exitCode=0
Jul 27 17:53:51.145325 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:51.144958    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"
Jul 27 17:53:51.145325 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:51.144985    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" event={"ID":"dfd06ee3-6a5e-4466-b38b-01c4b872f86a","Type":"ContainerDied","Data":"25f58ce1082ef84c30a72c2db2e4518fece02600e9dad9621137ea6eb58974a9"}
Jul 27 17:53:51.145325 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:51.145067    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" event={"ID":"dfd06ee3-6a5e-4466-b38b-01c4b872f86a","Type":"ContainerDied","Data":"deea67bcfba6c54ae79cae7f1459b38e0ba2c286a5f6c66ae06da885af6d2e9d"}
Jul 27 17:53:51.145325 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:51.145090    2571 scope.go:117] "RemoveContainer" containerID="25f58ce1082ef84c30a72c2db2e4518fece02600e9dad9621137ea6eb58974a9"
Jul 27 17:53:51.154876 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:51.154859    2571 scope.go:117] "RemoveContainer" containerID="97648a8701a705bf756371cde08c96966110462a0e8b9573797f63ced5161894"
Jul 27 17:53:51.163105 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:51.163089    2571 scope.go:117] "RemoveContainer" containerID="25f58ce1082ef84c30a72c2db2e4518fece02600e9dad9621137ea6eb58974a9"
Jul 27 17:53:51.163345 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:53:51.163327    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f58ce1082ef84c30a72c2db2e4518fece02600e9dad9621137ea6eb58974a9\": container with ID starting with 25f58ce1082ef84c30a72c2db2e4518fece02600e9dad9621137ea6eb58974a9 not found: ID does not exist" containerID="25f58ce1082ef84c30a72c2db2e4518fece02600e9dad9621137ea6eb58974a9"
Jul 27 17:53:51.163396 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:51.163353    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f58ce1082ef84c30a72c2db2e4518fece02600e9dad9621137ea6eb58974a9"} err="failed to get container status \"25f58ce1082ef84c30a72c2db2e4518fece02600e9dad9621137ea6eb58974a9\": rpc error: code = NotFound desc = could not find container \"25f58ce1082ef84c30a72c2db2e4518fece02600e9dad9621137ea6eb58974a9\": container with ID starting with 25f58ce1082ef84c30a72c2db2e4518fece02600e9dad9621137ea6eb58974a9 not found: ID does not exist"
Jul 27 17:53:51.163396 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:51.163370    2571 scope.go:117] "RemoveContainer" containerID="97648a8701a705bf756371cde08c96966110462a0e8b9573797f63ced5161894"
Jul 27 17:53:51.163632 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:53:51.163612    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97648a8701a705bf756371cde08c96966110462a0e8b9573797f63ced5161894\": container with ID starting with 97648a8701a705bf756371cde08c96966110462a0e8b9573797f63ced5161894 not found: ID does not exist" containerID="97648a8701a705bf756371cde08c96966110462a0e8b9573797f63ced5161894"
Jul 27 17:53:51.163678 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:51.163638    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97648a8701a705bf756371cde08c96966110462a0e8b9573797f63ced5161894"} err="failed to get container status \"97648a8701a705bf756371cde08c96966110462a0e8b9573797f63ced5161894\": rpc error: code = NotFound desc = could not find container \"97648a8701a705bf756371cde08c96966110462a0e8b9573797f63ced5161894\": container with ID starting with 97648a8701a705bf756371cde08c96966110462a0e8b9573797f63ced5161894 not found: ID does not exist"
Jul 27 17:53:51.168323 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:51.168303    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"]
Jul 27 17:53:51.172350 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:51.172328    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6"]
Jul 27 17:53:51.447560 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:51.447454    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-s6sn6" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 27 17:53:52.398765 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:52.398731    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" path="/var/lib/kubelet/pods/dfd06ee3-6a5e-4466-b38b-01c4b872f86a/volumes"
Jul 27 17:53:57.022573 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:53:57.022540    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"
Jul 27 17:54:03.610468 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.610393    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"]
Jul 27 17:54:03.610840 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.610729    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="storage-initializer"
Jul 27 17:54:03.610840 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.610742    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="storage-initializer"
Jul 27 17:54:03.610840 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.610750    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main"
Jul 27 17:54:03.610840 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.610756    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main"
Jul 27 17:54:03.610840 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.610815    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfd06ee3-6a5e-4466-b38b-01c4b872f86a" containerName="main"
Jul 27 17:54:03.615434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.615416    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.618376 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.618349    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 27 17:54:03.618538 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.618392    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 27 17:54:03.618538 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.618438    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-ms69c\""
Jul 27 17:54:03.618538 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.618444    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 27 17:54:03.618538 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.618397    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Jul 27 17:54:03.623668 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.623647    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"]
Jul 27 17:54:03.752077 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.752040    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k25jg\" (UniqueName: \"kubernetes.io/projected/ce3dc384-799f-48bd-9be1-371ef08c9887-kube-api-access-k25jg\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.752278 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.752109    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3dc384-799f-48bd-9be1-371ef08c9887-tls-certs\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.752278 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.752131    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-kserve-provision-location\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.752278 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.752150    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-home\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.752278 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.752167    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-dshm\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.752278 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.752253    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-model-cache\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.752278 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.752280    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-tmp-dir\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.853135 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.853099    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3dc384-799f-48bd-9be1-371ef08c9887-tls-certs\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.853135 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.853136    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-kserve-provision-location\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.853409 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.853166    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-home\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.853409 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.853190    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-dshm\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.853409 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.853251    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-model-cache\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.853409 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.853285    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-tmp-dir\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.853409 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.853316    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k25jg\" (UniqueName: \"kubernetes.io/projected/ce3dc384-799f-48bd-9be1-371ef08c9887-kube-api-access-k25jg\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.853691 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.853565    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-kserve-provision-location\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.853691 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.853657    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-model-cache\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.853691 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.853656    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-home\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.853807 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.853663    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-tmp-dir\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.855411 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.855390    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-dshm\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.855826 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.855804    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3dc384-799f-48bd-9be1-371ef08c9887-tls-certs\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.862333 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.862280    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k25jg\" (UniqueName: \"kubernetes.io/projected/ce3dc384-799f-48bd-9be1-371ef08c9887-kube-api-access-k25jg\") pod \"auth-disabled-test-kserve-6f468f6548-j7b8s\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:03.926528 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:03.926480    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:04.059174 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:04.059151    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"]
Jul 27 17:54:04.061534 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:54:04.061489    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3dc384_799f_48bd_9be1_371ef08c9887.slice/crio-f58c263a690b19d9657d29dfbde66c3725323e5fea0b2650ae272b8a53f1f56b WatchSource:0}: Error finding container f58c263a690b19d9657d29dfbde66c3725323e5fea0b2650ae272b8a53f1f56b: Status 404 returned error can't find the container with id f58c263a690b19d9657d29dfbde66c3725323e5fea0b2650ae272b8a53f1f56b
Jul 27 17:54:04.209258 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:04.209221    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" event={"ID":"ce3dc384-799f-48bd-9be1-371ef08c9887","Type":"ContainerStarted","Data":"ff742d6aed8e4b6728125b5104a0c9f589a4fd9e0af1875a4f6ee653f8118d80"}
Jul 27 17:54:04.209419 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:04.209273    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" event={"ID":"ce3dc384-799f-48bd-9be1-371ef08c9887","Type":"ContainerStarted","Data":"f58c263a690b19d9657d29dfbde66c3725323e5fea0b2650ae272b8a53f1f56b"}
Jul 27 17:54:12.256795 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:12.256757    2571 generic.go:356] "Generic (PLEG): container finished" podID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerID="ff742d6aed8e4b6728125b5104a0c9f589a4fd9e0af1875a4f6ee653f8118d80" exitCode=0
Jul 27 17:54:12.257205 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:12.256830    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" event={"ID":"ce3dc384-799f-48bd-9be1-371ef08c9887","Type":"ContainerDied","Data":"ff742d6aed8e4b6728125b5104a0c9f589a4fd9e0af1875a4f6ee653f8118d80"}
Jul 27 17:54:13.264118 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:13.264080    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" event={"ID":"ce3dc384-799f-48bd-9be1-371ef08c9887","Type":"ContainerStarted","Data":"463df5b2c48b27797d2bf662e7bf9ff3a0379ab98953eb09011b6010d2189179"}
Jul 27 17:54:13.287457 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:13.287411    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podStartSLOduration=3.085445309 podStartE2EDuration="10.287396543s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.201951234s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:54:13.284527317 +0000 UTC m=+1535.329265406" watchObservedRunningTime="2026-07-27 17:54:13.287396543 +0000 UTC m=+1535.332134681"
Jul 27 17:54:13.927114 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:13.927081    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:13.927339 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:13.927226    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:54:13.928757 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:13.928726    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 27 17:54:20.657694 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:20.657657    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"]
Jul 27 17:54:23.927469 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:23.927423    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 27 17:54:27.022850 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:27.022805    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl" podUID="7bc9fb26-e200-4452-83fe-43f15ba5e2da" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:54:33.928011 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:33.927965    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 27 17:54:35.677370 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:35.677316    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl" podUID="7bc9fb26-e200-4452-83fe-43f15ba5e2da" containerName="main" containerID="cri-o://e3121f0861f91ef2523ba270a8e2b6c8b9eb8a612bbd41c86aa694c53d21caca" gracePeriod=45
Jul 27 17:54:35.918118 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:35.918088    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"
Jul 27 17:54:36.040551 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.040508    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wskwd\" (UniqueName: \"kubernetes.io/projected/7bc9fb26-e200-4452-83fe-43f15ba5e2da-kube-api-access-wskwd\") pod \"7bc9fb26-e200-4452-83fe-43f15ba5e2da\" (UID: \"7bc9fb26-e200-4452-83fe-43f15ba5e2da\") "
Jul 27 17:54:36.040745 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.040645    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc9fb26-e200-4452-83fe-43f15ba5e2da-tls-certs\") pod \"7bc9fb26-e200-4452-83fe-43f15ba5e2da\" (UID: \"7bc9fb26-e200-4452-83fe-43f15ba5e2da\") "
Jul 27 17:54:36.042698 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.042672    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc9fb26-e200-4452-83fe-43f15ba5e2da-kube-api-access-wskwd" (OuterVolumeSpecName: "kube-api-access-wskwd") pod "7bc9fb26-e200-4452-83fe-43f15ba5e2da" (UID: "7bc9fb26-e200-4452-83fe-43f15ba5e2da"). InnerVolumeSpecName "kube-api-access-wskwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:54:36.042859 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.042839    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc9fb26-e200-4452-83fe-43f15ba5e2da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7bc9fb26-e200-4452-83fe-43f15ba5e2da" (UID: "7bc9fb26-e200-4452-83fe-43f15ba5e2da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:54:36.141681 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.141638    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wskwd\" (UniqueName: \"kubernetes.io/projected/7bc9fb26-e200-4452-83fe-43f15ba5e2da-kube-api-access-wskwd\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:54:36.141681 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.141679    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc9fb26-e200-4452-83fe-43f15ba5e2da-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:54:36.392162 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.392079    2571 generic.go:356] "Generic (PLEG): container finished" podID="7bc9fb26-e200-4452-83fe-43f15ba5e2da" containerID="e3121f0861f91ef2523ba270a8e2b6c8b9eb8a612bbd41c86aa694c53d21caca" exitCode=0
Jul 27 17:54:36.392162 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.392125    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"
Jul 27 17:54:36.392350 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.392175    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl" event={"ID":"7bc9fb26-e200-4452-83fe-43f15ba5e2da","Type":"ContainerDied","Data":"e3121f0861f91ef2523ba270a8e2b6c8b9eb8a612bbd41c86aa694c53d21caca"}
Jul 27 17:54:36.392350 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.392239    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl" event={"ID":"7bc9fb26-e200-4452-83fe-43f15ba5e2da","Type":"ContainerDied","Data":"f63b43c41dc23fe74fd6a6a3368e768869d5c1abe07bf83cc7ab3a4a58f070fc"}
Jul 27 17:54:36.392350 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.392274    2571 scope.go:117] "RemoveContainer" containerID="e3121f0861f91ef2523ba270a8e2b6c8b9eb8a612bbd41c86aa694c53d21caca"
Jul 27 17:54:36.402583 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.402560    2571 scope.go:117] "RemoveContainer" containerID="e3121f0861f91ef2523ba270a8e2b6c8b9eb8a612bbd41c86aa694c53d21caca"
Jul 27 17:54:36.402941 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:54:36.402919    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3121f0861f91ef2523ba270a8e2b6c8b9eb8a612bbd41c86aa694c53d21caca\": container with ID starting with e3121f0861f91ef2523ba270a8e2b6c8b9eb8a612bbd41c86aa694c53d21caca not found: ID does not exist" containerID="e3121f0861f91ef2523ba270a8e2b6c8b9eb8a612bbd41c86aa694c53d21caca"
Jul 27 17:54:36.403023 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.402948    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3121f0861f91ef2523ba270a8e2b6c8b9eb8a612bbd41c86aa694c53d21caca"} err="failed to get container status \"e3121f0861f91ef2523ba270a8e2b6c8b9eb8a612bbd41c86aa694c53d21caca\": rpc error: code = NotFound desc = could not find container \"e3121f0861f91ef2523ba270a8e2b6c8b9eb8a612bbd41c86aa694c53d21caca\": container with ID starting with e3121f0861f91ef2523ba270a8e2b6c8b9eb8a612bbd41c86aa694c53d21caca not found: ID does not exist"
Jul 27 17:54:36.416690 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.416664    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"]
Jul 27 17:54:36.420301 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:36.420276    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche97mnl"]
Jul 27 17:54:38.399489 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:38.399443    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bc9fb26-e200-4452-83fe-43f15ba5e2da" path="/var/lib/kubelet/pods/7bc9fb26-e200-4452-83fe-43f15ba5e2da/volumes"
Jul 27 17:54:41.187768 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.187731    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"]
Jul 27 17:54:41.188159 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.188042    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc9fb26-e200-4452-83fe-43f15ba5e2da" containerName="main"
Jul 27 17:54:41.188159 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.188055    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc9fb26-e200-4452-83fe-43f15ba5e2da" containerName="main"
Jul 27 17:54:41.188159 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.188116    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc9fb26-e200-4452-83fe-43f15ba5e2da" containerName="main"
Jul 27 17:54:41.191010 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.190994    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"
Jul 27 17:54:41.198605 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.198582    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-kfjlv\""
Jul 27 17:54:41.198742 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.198589    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 27 17:54:41.198742 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.198626    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 27 17:54:41.198742 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.198612    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 27 17:54:41.198957 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.198942    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-kj2lx\""
Jul 27 17:54:41.203948 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.203926    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"]
Jul 27 17:54:41.286604 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.286567    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbsw\" (UniqueName: \"kubernetes.io/projected/51e5479c-debb-4883-b4b6-04cb9ad73ea2-kube-api-access-cwbsw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj\" (UID: \"51e5479c-debb-4883-b4b6-04cb9ad73ea2\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"
Jul 27 17:54:41.286789 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.286632    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e5479c-debb-4883-b4b6-04cb9ad73ea2-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj\" (UID: \"51e5479c-debb-4883-b4b6-04cb9ad73ea2\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"
Jul 27 17:54:41.388138 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.388088    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbsw\" (UniqueName: \"kubernetes.io/projected/51e5479c-debb-4883-b4b6-04cb9ad73ea2-kube-api-access-cwbsw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj\" (UID: \"51e5479c-debb-4883-b4b6-04cb9ad73ea2\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"
Jul 27 17:54:41.388343 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.388176    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e5479c-debb-4883-b4b6-04cb9ad73ea2-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj\" (UID: \"51e5479c-debb-4883-b4b6-04cb9ad73ea2\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"
Jul 27 17:54:41.390775 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.390751    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e5479c-debb-4883-b4b6-04cb9ad73ea2-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj\" (UID: \"51e5479c-debb-4883-b4b6-04cb9ad73ea2\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"
Jul 27 17:54:41.398105 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.398074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbsw\" (UniqueName: \"kubernetes.io/projected/51e5479c-debb-4883-b4b6-04cb9ad73ea2-kube-api-access-cwbsw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj\" (UID: \"51e5479c-debb-4883-b4b6-04cb9ad73ea2\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"
Jul 27 17:54:41.501643 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.501607    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"
Jul 27 17:54:41.653664 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:41.653638    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"]
Jul 27 17:54:41.655956 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:54:41.655922    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e5479c_debb_4883_b4b6_04cb9ad73ea2.slice/crio-c85dfb4271fbd6ac13ad2ca6d0db33a0c909a74db3dcf97930af43e3189b269c WatchSource:0}: Error finding container c85dfb4271fbd6ac13ad2ca6d0db33a0c909a74db3dcf97930af43e3189b269c: Status 404 returned error can't find the container with id c85dfb4271fbd6ac13ad2ca6d0db33a0c909a74db3dcf97930af43e3189b269c
Jul 27 17:54:42.427669 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:42.427637    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj" event={"ID":"51e5479c-debb-4883-b4b6-04cb9ad73ea2","Type":"ContainerStarted","Data":"401c5796cee2526217404b7788d08f71bc5683b6aa2a4531420910a130d8732b"}
Jul 27 17:54:42.428122 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:42.427687    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj" event={"ID":"51e5479c-debb-4883-b4b6-04cb9ad73ea2","Type":"ContainerStarted","Data":"c85dfb4271fbd6ac13ad2ca6d0db33a0c909a74db3dcf97930af43e3189b269c"}
Jul 27 17:54:42.428122 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:42.427747    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"
Jul 27 17:54:42.446797 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:42.446750    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj" podStartSLOduration=1.446734767 podStartE2EDuration="1.446734767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:54:42.446203289 +0000 UTC m=+1564.490941371" watchObservedRunningTime="2026-07-27 17:54:42.446734767 +0000 UTC m=+1564.491472847"
Jul 27 17:54:43.927674 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:43.927624    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 27 17:54:53.927349 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:54:53.927299    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 27 17:55:03.927172 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:03.927126    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 27 17:55:13.434792 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:13.434756    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"
Jul 27 17:55:13.927513 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:13.927455    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 27 17:55:14.544071 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:14.544038    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"]
Jul 27 17:55:23.436022 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:23.435952    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj" podUID="51e5479c-debb-4883-b4b6-04cb9ad73ea2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:55:23.927780 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:23.927723    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 27 17:55:29.562955 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:29.562905    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj" podUID="51e5479c-debb-4883-b4b6-04cb9ad73ea2" containerName="main" containerID="cri-o://401c5796cee2526217404b7788d08f71bc5683b6aa2a4531420910a130d8732b" gracePeriod=45
Jul 27 17:55:29.806846 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:29.806821    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"
Jul 27 17:55:29.910613 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:29.910518    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e5479c-debb-4883-b4b6-04cb9ad73ea2-tls-certs\") pod \"51e5479c-debb-4883-b4b6-04cb9ad73ea2\" (UID: \"51e5479c-debb-4883-b4b6-04cb9ad73ea2\") "
Jul 27 17:55:29.910613 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:29.910582    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwbsw\" (UniqueName: \"kubernetes.io/projected/51e5479c-debb-4883-b4b6-04cb9ad73ea2-kube-api-access-cwbsw\") pod \"51e5479c-debb-4883-b4b6-04cb9ad73ea2\" (UID: \"51e5479c-debb-4883-b4b6-04cb9ad73ea2\") "
Jul 27 17:55:29.912769 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:29.912735    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e5479c-debb-4883-b4b6-04cb9ad73ea2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51e5479c-debb-4883-b4b6-04cb9ad73ea2" (UID: "51e5479c-debb-4883-b4b6-04cb9ad73ea2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:55:29.912887 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:29.912765    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e5479c-debb-4883-b4b6-04cb9ad73ea2-kube-api-access-cwbsw" (OuterVolumeSpecName: "kube-api-access-cwbsw") pod "51e5479c-debb-4883-b4b6-04cb9ad73ea2" (UID: "51e5479c-debb-4883-b4b6-04cb9ad73ea2"). InnerVolumeSpecName "kube-api-access-cwbsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:55:30.012026 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:30.011992    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e5479c-debb-4883-b4b6-04cb9ad73ea2-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:55:30.012026 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:30.012020    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwbsw\" (UniqueName: \"kubernetes.io/projected/51e5479c-debb-4883-b4b6-04cb9ad73ea2-kube-api-access-cwbsw\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:55:30.681712 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:30.681581    2571 generic.go:356] "Generic (PLEG): container finished" podID="51e5479c-debb-4883-b4b6-04cb9ad73ea2" containerID="401c5796cee2526217404b7788d08f71bc5683b6aa2a4531420910a130d8732b" exitCode=0
Jul 27 17:55:30.681712 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:30.681641    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"
Jul 27 17:55:30.682276 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:30.681638    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj" event={"ID":"51e5479c-debb-4883-b4b6-04cb9ad73ea2","Type":"ContainerDied","Data":"401c5796cee2526217404b7788d08f71bc5683b6aa2a4531420910a130d8732b"}
Jul 27 17:55:30.682276 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:30.681810    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj" event={"ID":"51e5479c-debb-4883-b4b6-04cb9ad73ea2","Type":"ContainerDied","Data":"c85dfb4271fbd6ac13ad2ca6d0db33a0c909a74db3dcf97930af43e3189b269c"}
Jul 27 17:55:30.682276 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:30.681845    2571 scope.go:117] "RemoveContainer" containerID="401c5796cee2526217404b7788d08f71bc5683b6aa2a4531420910a130d8732b"
Jul 27 17:55:30.690722 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:30.690700    2571 scope.go:117] "RemoveContainer" containerID="401c5796cee2526217404b7788d08f71bc5683b6aa2a4531420910a130d8732b"
Jul 27 17:55:30.690960 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:55:30.690940    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401c5796cee2526217404b7788d08f71bc5683b6aa2a4531420910a130d8732b\": container with ID starting with 401c5796cee2526217404b7788d08f71bc5683b6aa2a4531420910a130d8732b not found: ID does not exist" containerID="401c5796cee2526217404b7788d08f71bc5683b6aa2a4531420910a130d8732b"
Jul 27 17:55:30.691040 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:30.690966    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401c5796cee2526217404b7788d08f71bc5683b6aa2a4531420910a130d8732b"} err="failed to get container status \"401c5796cee2526217404b7788d08f71bc5683b6aa2a4531420910a130d8732b\": rpc error: code = NotFound desc = could not find container \"401c5796cee2526217404b7788d08f71bc5683b6aa2a4531420910a130d8732b\": container with ID starting with 401c5796cee2526217404b7788d08f71bc5683b6aa2a4531420910a130d8732b not found: ID does not exist"
Jul 27 17:55:30.698523 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:30.698477    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"]
Jul 27 17:55:30.704604 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:30.704579    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587w68fj"]
Jul 27 17:55:32.399397 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:32.399352    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51e5479c-debb-4883-b4b6-04cb9ad73ea2" path="/var/lib/kubelet/pods/51e5479c-debb-4883-b4b6-04cb9ad73ea2/volumes"
Jul 27 17:55:33.928010 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:33.927971    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 27 17:55:42.997242 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:42.997191    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"]
Jul 27 17:55:42.997803 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:42.997627    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e5479c-debb-4883-b4b6-04cb9ad73ea2" containerName="main"
Jul 27 17:55:42.997803 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:42.997647    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e5479c-debb-4883-b4b6-04cb9ad73ea2" containerName="main"
Jul 27 17:55:42.997803 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:42.997767    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e5479c-debb-4883-b4b6-04cb9ad73ea2" containerName="main"
Jul 27 17:55:43.001003 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.000979    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"
Jul 27 17:55:43.004887 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.004799    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 27 17:55:43.005022 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.004947    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-flx8t\""
Jul 27 17:55:43.005022 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.004955    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 27 17:55:43.005022 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.004799    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-96z8p\""
Jul 27 17:55:43.005234 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.005066    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 27 17:55:43.009027 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.008634    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"]
Jul 27 17:55:43.029051 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.029022    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knkqc\" (UniqueName: \"kubernetes.io/projected/f0faa444-e66e-4cef-83e9-1205651a2c0c-kube-api-access-knkqc\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb\" (UID: \"f0faa444-e66e-4cef-83e9-1205651a2c0c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"
Jul 27 17:55:43.029213 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.029090    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0faa444-e66e-4cef-83e9-1205651a2c0c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb\" (UID: \"f0faa444-e66e-4cef-83e9-1205651a2c0c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"
Jul 27 17:55:43.130520 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.130436    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkqc\" (UniqueName: \"kubernetes.io/projected/f0faa444-e66e-4cef-83e9-1205651a2c0c-kube-api-access-knkqc\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb\" (UID: \"f0faa444-e66e-4cef-83e9-1205651a2c0c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"
Jul 27 17:55:43.130731 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.130587    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0faa444-e66e-4cef-83e9-1205651a2c0c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb\" (UID: \"f0faa444-e66e-4cef-83e9-1205651a2c0c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"
Jul 27 17:55:43.133235 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.133205    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0faa444-e66e-4cef-83e9-1205651a2c0c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb\" (UID: \"f0faa444-e66e-4cef-83e9-1205651a2c0c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"
Jul 27 17:55:43.140030 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.140008    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knkqc\" (UniqueName: \"kubernetes.io/projected/f0faa444-e66e-4cef-83e9-1205651a2c0c-kube-api-access-knkqc\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb\" (UID: \"f0faa444-e66e-4cef-83e9-1205651a2c0c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"
Jul 27 17:55:43.312690 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.312597    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"
Jul 27 17:55:43.458885 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.458856    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"]
Jul 27 17:55:43.461625 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:55:43.461586    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0faa444_e66e_4cef_83e9_1205651a2c0c.slice/crio-d256ee21c18b8f7e771ada24439dc92a5c6844ed267eab87c5f9ea31987e1f46 WatchSource:0}: Error finding container d256ee21c18b8f7e771ada24439dc92a5c6844ed267eab87c5f9ea31987e1f46: Status 404 returned error can't find the container with id d256ee21c18b8f7e771ada24439dc92a5c6844ed267eab87c5f9ea31987e1f46
Jul 27 17:55:43.759064 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.759025    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb" event={"ID":"f0faa444-e66e-4cef-83e9-1205651a2c0c","Type":"ContainerStarted","Data":"2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471"}
Jul 27 17:55:43.759290 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.759082    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb" event={"ID":"f0faa444-e66e-4cef-83e9-1205651a2c0c","Type":"ContainerStarted","Data":"d256ee21c18b8f7e771ada24439dc92a5c6844ed267eab87c5f9ea31987e1f46"}
Jul 27 17:55:43.759290 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.759167    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"
Jul 27 17:55:43.927040 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:43.926995    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 27 17:55:53.927335 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:55:53.927285    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 27 17:56:03.937079 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:03.937049    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:56:03.944843 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:03.944820    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:56:03.962480 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:03.962432    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb" podStartSLOduration=21.96241938 podStartE2EDuration="21.96241938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:55:43.785277028 +0000 UTC m=+1625.830015111" watchObservedRunningTime="2026-07-27 17:56:03.96241938 +0000 UTC m=+1646.007157460"
Jul 27 17:56:08.528678 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:08.528645    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"]
Jul 27 17:56:14.766555 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:14.766527    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"
Jul 27 17:56:16.089288 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:16.089253    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"]
Jul 27 17:56:23.550887 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:23.550810    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main" containerID="cri-o://463df5b2c48b27797d2bf662e7bf9ff3a0379ab98953eb09011b6010d2189179" gracePeriod=45
Jul 27 17:56:23.937836 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:23.937729    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 27 17:56:24.295211 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.295187    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:56:24.390764 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.390724    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-kserve-provision-location\") pod \"ce3dc384-799f-48bd-9be1-371ef08c9887\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") "
Jul 27 17:56:24.390764 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.390762    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-home\") pod \"ce3dc384-799f-48bd-9be1-371ef08c9887\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") "
Jul 27 17:56:24.391026 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.390789    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-model-cache\") pod \"ce3dc384-799f-48bd-9be1-371ef08c9887\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") "
Jul 27 17:56:24.391026 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.390823    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3dc384-799f-48bd-9be1-371ef08c9887-tls-certs\") pod \"ce3dc384-799f-48bd-9be1-371ef08c9887\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") "
Jul 27 17:56:24.391026 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.390838    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-dshm\") pod \"ce3dc384-799f-48bd-9be1-371ef08c9887\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") "
Jul 27 17:56:24.391026 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.390888    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k25jg\" (UniqueName: \"kubernetes.io/projected/ce3dc384-799f-48bd-9be1-371ef08c9887-kube-api-access-k25jg\") pod \"ce3dc384-799f-48bd-9be1-371ef08c9887\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") "
Jul 27 17:56:24.391026 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.390930    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-tmp-dir\") pod \"ce3dc384-799f-48bd-9be1-371ef08c9887\" (UID: \"ce3dc384-799f-48bd-9be1-371ef08c9887\") "
Jul 27 17:56:24.391309 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.391085    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce3dc384-799f-48bd-9be1-371ef08c9887" (UID: "ce3dc384-799f-48bd-9be1-371ef08c9887"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:24.391309 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.391220    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:56:24.391606 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.391576    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-home" (OuterVolumeSpecName: "home") pod "ce3dc384-799f-48bd-9be1-371ef08c9887" (UID: "ce3dc384-799f-48bd-9be1-371ef08c9887"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:24.393750 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.393719    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3dc384-799f-48bd-9be1-371ef08c9887-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce3dc384-799f-48bd-9be1-371ef08c9887" (UID: "ce3dc384-799f-48bd-9be1-371ef08c9887"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:56:24.393872 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.393746    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3dc384-799f-48bd-9be1-371ef08c9887-kube-api-access-k25jg" (OuterVolumeSpecName: "kube-api-access-k25jg") pod "ce3dc384-799f-48bd-9be1-371ef08c9887" (UID: "ce3dc384-799f-48bd-9be1-371ef08c9887"). InnerVolumeSpecName "kube-api-access-k25jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:56:24.394042 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.394013    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-dshm" (OuterVolumeSpecName: "dshm") pod "ce3dc384-799f-48bd-9be1-371ef08c9887" (UID: "ce3dc384-799f-48bd-9be1-371ef08c9887"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:24.403985 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.403944    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce3dc384-799f-48bd-9be1-371ef08c9887" (UID: "ce3dc384-799f-48bd-9be1-371ef08c9887"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:24.460056 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.460022    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce3dc384-799f-48bd-9be1-371ef08c9887" (UID: "ce3dc384-799f-48bd-9be1-371ef08c9887"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:24.492157 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.492094    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k25jg\" (UniqueName: \"kubernetes.io/projected/ce3dc384-799f-48bd-9be1-371ef08c9887-kube-api-access-k25jg\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:56:24.492157 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.492119    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:56:24.492157 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.492128    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-kserve-provision-location\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:56:24.492157 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.492138    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:56:24.492157 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.492146    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3dc384-799f-48bd-9be1-371ef08c9887-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:56:24.492157 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.492153    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce3dc384-799f-48bd-9be1-371ef08c9887-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:56:24.766078 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.765994    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb" podUID="f0faa444-e66e-4cef-83e9-1205651a2c0c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:56:24.969070 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.969038    2571 generic.go:356] "Generic (PLEG): container finished" podID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerID="463df5b2c48b27797d2bf662e7bf9ff3a0379ab98953eb09011b6010d2189179" exitCode=0
Jul 27 17:56:24.969232 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.969090    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"
Jul 27 17:56:24.969232 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.969126    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" event={"ID":"ce3dc384-799f-48bd-9be1-371ef08c9887","Type":"ContainerDied","Data":"463df5b2c48b27797d2bf662e7bf9ff3a0379ab98953eb09011b6010d2189179"}
Jul 27 17:56:24.969232 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.969184    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s" event={"ID":"ce3dc384-799f-48bd-9be1-371ef08c9887","Type":"ContainerDied","Data":"f58c263a690b19d9657d29dfbde66c3725323e5fea0b2650ae272b8a53f1f56b"}
Jul 27 17:56:24.969232 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.969209    2571 scope.go:117] "RemoveContainer" containerID="463df5b2c48b27797d2bf662e7bf9ff3a0379ab98953eb09011b6010d2189179"
Jul 27 17:56:24.980611 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.980592    2571 scope.go:117] "RemoveContainer" containerID="ff742d6aed8e4b6728125b5104a0c9f589a4fd9e0af1875a4f6ee653f8118d80"
Jul 27 17:56:24.988541 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.988521    2571 scope.go:117] "RemoveContainer" containerID="463df5b2c48b27797d2bf662e7bf9ff3a0379ab98953eb09011b6010d2189179"
Jul 27 17:56:24.988780 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:56:24.988761    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"463df5b2c48b27797d2bf662e7bf9ff3a0379ab98953eb09011b6010d2189179\": container with ID starting with 463df5b2c48b27797d2bf662e7bf9ff3a0379ab98953eb09011b6010d2189179 not found: ID does not exist" containerID="463df5b2c48b27797d2bf662e7bf9ff3a0379ab98953eb09011b6010d2189179"
Jul 27 17:56:24.988836 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.988788    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"463df5b2c48b27797d2bf662e7bf9ff3a0379ab98953eb09011b6010d2189179"} err="failed to get container status \"463df5b2c48b27797d2bf662e7bf9ff3a0379ab98953eb09011b6010d2189179\": rpc error: code = NotFound desc = could not find container \"463df5b2c48b27797d2bf662e7bf9ff3a0379ab98953eb09011b6010d2189179\": container with ID starting with 463df5b2c48b27797d2bf662e7bf9ff3a0379ab98953eb09011b6010d2189179 not found: ID does not exist"
Jul 27 17:56:24.988836 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.988809    2571 scope.go:117] "RemoveContainer" containerID="ff742d6aed8e4b6728125b5104a0c9f589a4fd9e0af1875a4f6ee653f8118d80"
Jul 27 17:56:24.989033 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:56:24.989014    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff742d6aed8e4b6728125b5104a0c9f589a4fd9e0af1875a4f6ee653f8118d80\": container with ID starting with ff742d6aed8e4b6728125b5104a0c9f589a4fd9e0af1875a4f6ee653f8118d80 not found: ID does not exist" containerID="ff742d6aed8e4b6728125b5104a0c9f589a4fd9e0af1875a4f6ee653f8118d80"
Jul 27 17:56:24.989090 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.989043    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff742d6aed8e4b6728125b5104a0c9f589a4fd9e0af1875a4f6ee653f8118d80"} err="failed to get container status \"ff742d6aed8e4b6728125b5104a0c9f589a4fd9e0af1875a4f6ee653f8118d80\": rpc error: code = NotFound desc = could not find container \"ff742d6aed8e4b6728125b5104a0c9f589a4fd9e0af1875a4f6ee653f8118d80\": container with ID starting with ff742d6aed8e4b6728125b5104a0c9f589a4fd9e0af1875a4f6ee653f8118d80 not found: ID does not exist"
Jul 27 17:56:24.995580 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.995555    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"]
Jul 27 17:56:24.998542 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:24.998521    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-j7b8s"]
Jul 27 17:56:26.398621 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:26.398589    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" path="/var/lib/kubelet/pods/ce3dc384-799f-48bd-9be1-371ef08c9887/volumes"
Jul 27 17:56:27.721549 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:27.721490    2571 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb" message="" err="command '/bin/sleep 15' exited with 137: "
Jul 27 17:56:27.721927 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:56:27.721548    2571 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb" podUID="f0faa444-e66e-4cef-83e9-1205651a2c0c" containerName="main" containerID="cri-o://2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471"
Jul 27 17:56:27.721927 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:27.721602    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb" podUID="f0faa444-e66e-4cef-83e9-1205651a2c0c" containerName="main" containerID="cri-o://2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471" gracePeriod=49
Jul 27 17:56:27.865059 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:27.865036    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb_f0faa444-e66e-4cef-83e9-1205651a2c0c/main/0.log"
Jul 27 17:56:27.867363 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:27.867348    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"
Jul 27 17:56:27.987208 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:27.987125    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb_f0faa444-e66e-4cef-83e9-1205651a2c0c/main/0.log"
Jul 27 17:56:27.987208 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:27.987166    2571 generic.go:356] "Generic (PLEG): container finished" podID="f0faa444-e66e-4cef-83e9-1205651a2c0c" containerID="2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471" exitCode=1
Jul 27 17:56:27.987401 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:27.987222    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"
Jul 27 17:56:27.987401 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:27.987260    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb" event={"ID":"f0faa444-e66e-4cef-83e9-1205651a2c0c","Type":"ContainerDied","Data":"2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471"}
Jul 27 17:56:27.987401 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:27.987334    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb" event={"ID":"f0faa444-e66e-4cef-83e9-1205651a2c0c","Type":"ContainerDied","Data":"d256ee21c18b8f7e771ada24439dc92a5c6844ed267eab87c5f9ea31987e1f46"}
Jul 27 17:56:27.987401 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:27.987358    2571 scope.go:117] "RemoveContainer" containerID="2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471"
Jul 27 17:56:27.995810 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:27.995793    2571 scope.go:117] "RemoveContainer" containerID="2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471"
Jul 27 17:56:27.996049 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:56:27.996032    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471\": container with ID starting with 2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471 not found: ID does not exist" containerID="2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471"
Jul 27 17:56:27.996122 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:27.996056    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471"} err="failed to get container status \"2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471\": rpc error: code = NotFound desc = could not find container \"2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471\": container with ID starting with 2362b17129d828a1853745174c735f4640ce0b42de59dbd945c0a6ac5f1af471 not found: ID does not exist"
Jul 27 17:56:28.021394 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:28.021364    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0faa444-e66e-4cef-83e9-1205651a2c0c-tls-certs\") pod \"f0faa444-e66e-4cef-83e9-1205651a2c0c\" (UID: \"f0faa444-e66e-4cef-83e9-1205651a2c0c\") "
Jul 27 17:56:28.021528 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:28.021435    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knkqc\" (UniqueName: \"kubernetes.io/projected/f0faa444-e66e-4cef-83e9-1205651a2c0c-kube-api-access-knkqc\") pod \"f0faa444-e66e-4cef-83e9-1205651a2c0c\" (UID: \"f0faa444-e66e-4cef-83e9-1205651a2c0c\") "
Jul 27 17:56:28.023528 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:28.023480    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0faa444-e66e-4cef-83e9-1205651a2c0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0faa444-e66e-4cef-83e9-1205651a2c0c" (UID: "f0faa444-e66e-4cef-83e9-1205651a2c0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:56:28.023627 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:28.023569    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0faa444-e66e-4cef-83e9-1205651a2c0c-kube-api-access-knkqc" (OuterVolumeSpecName: "kube-api-access-knkqc") pod "f0faa444-e66e-4cef-83e9-1205651a2c0c" (UID: "f0faa444-e66e-4cef-83e9-1205651a2c0c"). InnerVolumeSpecName "kube-api-access-knkqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:56:28.122167 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:28.122133    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knkqc\" (UniqueName: \"kubernetes.io/projected/f0faa444-e66e-4cef-83e9-1205651a2c0c-kube-api-access-knkqc\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:56:28.122167 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:28.122163    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0faa444-e66e-4cef-83e9-1205651a2c0c-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:56:28.310011 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:28.309978    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"]
Jul 27 17:56:28.314913 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:28.314888    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfz2nkb"]
Jul 27 17:56:28.399033 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:28.398997    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0faa444-e66e-4cef-83e9-1205651a2c0c" path="/var/lib/kubelet/pods/f0faa444-e66e-4cef-83e9-1205651a2c0c/volumes"
Jul 27 17:56:48.517515 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.517471    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"]
Jul 27 17:56:48.517886 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.517791    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="storage-initializer"
Jul 27 17:56:48.517886 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.517803    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="storage-initializer"
Jul 27 17:56:48.517886 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.517824    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0faa444-e66e-4cef-83e9-1205651a2c0c" containerName="main"
Jul 27 17:56:48.517886 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.517830    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0faa444-e66e-4cef-83e9-1205651a2c0c" containerName="main"
Jul 27 17:56:48.517886 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.517837    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main"
Jul 27 17:56:48.517886 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.517843    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main"
Jul 27 17:56:48.518084 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.517900    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0faa444-e66e-4cef-83e9-1205651a2c0c" containerName="main"
Jul 27 17:56:48.518084 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.517909    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce3dc384-799f-48bd-9be1-371ef08c9887" containerName="main"
Jul 27 17:56:48.520723 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.520708    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"
Jul 27 17:56:48.525754 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.525725    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 27 17:56:48.525754 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.525742    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-xm4b8\""
Jul 27 17:56:48.525948 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.525770    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-epp-sa-dockercfg-p8hv9\""
Jul 27 17:56:48.525948 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.525875    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 27 17:56:48.526055 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.525953    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 27 17:56:48.529739 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.529717    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"]
Jul 27 17:56:48.573628 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.573595    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23131789-0408-45db-969c-7ddcdeba9d07-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m\" (UID: \"23131789-0408-45db-969c-7ddcdeba9d07\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"
Jul 27 17:56:48.573802 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.573638    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd8l4\" (UniqueName: \"kubernetes.io/projected/23131789-0408-45db-969c-7ddcdeba9d07-kube-api-access-zd8l4\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m\" (UID: \"23131789-0408-45db-969c-7ddcdeba9d07\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"
Jul 27 17:56:48.674918 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.674885    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd8l4\" (UniqueName: \"kubernetes.io/projected/23131789-0408-45db-969c-7ddcdeba9d07-kube-api-access-zd8l4\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m\" (UID: \"23131789-0408-45db-969c-7ddcdeba9d07\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"
Jul 27 17:56:48.675101 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.675011    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23131789-0408-45db-969c-7ddcdeba9d07-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m\" (UID: \"23131789-0408-45db-969c-7ddcdeba9d07\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"
Jul 27 17:56:48.677814 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.677788    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23131789-0408-45db-969c-7ddcdeba9d07-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m\" (UID: \"23131789-0408-45db-969c-7ddcdeba9d07\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"
Jul 27 17:56:48.682825 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.682803    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd8l4\" (UniqueName: \"kubernetes.io/projected/23131789-0408-45db-969c-7ddcdeba9d07-kube-api-access-zd8l4\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m\" (UID: \"23131789-0408-45db-969c-7ddcdeba9d07\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"
Jul 27 17:56:48.830436 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.830353    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"
Jul 27 17:56:48.971468 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:48.971433    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"]
Jul 27 17:56:48.974737 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:56:48.973565    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23131789_0408_45db_969c_7ddcdeba9d07.slice/crio-353f515b59171f8093f5537f3670a745b74f47a6527ac18e823d15c4739839f5 WatchSource:0}: Error finding container 353f515b59171f8093f5537f3670a745b74f47a6527ac18e823d15c4739839f5: Status 404 returned error can't find the container with id 353f515b59171f8093f5537f3670a745b74f47a6527ac18e823d15c4739839f5
Jul 27 17:56:49.090154 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:49.090052    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m" event={"ID":"23131789-0408-45db-969c-7ddcdeba9d07","Type":"ContainerStarted","Data":"ff345e52cf0519adc9232526f6313339772cab72efdbf6d9e66d6097c490d4f2"}
Jul 27 17:56:49.090154 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:49.090125    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m" event={"ID":"23131789-0408-45db-969c-7ddcdeba9d07","Type":"ContainerStarted","Data":"353f515b59171f8093f5537f3670a745b74f47a6527ac18e823d15c4739839f5"}
Jul 27 17:56:49.090409 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:49.090385    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"
Jul 27 17:56:49.112695 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:56:49.112637    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m" podStartSLOduration=1.112621682 podStartE2EDuration="1.112621682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:56:49.110695038 +0000 UTC m=+1691.155433138" watchObservedRunningTime="2026-07-27 17:56:49.112621682 +0000 UTC m=+1691.157359796"
Jul 27 17:57:09.293867 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.293791    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"]
Jul 27 17:57:09.297160 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.297142    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.302853 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.301783    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 27 17:57:09.302853 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.302144    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 27 17:57:09.302853 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.302438    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 27 17:57:09.302853 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.302585    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-6gpqv\""
Jul 27 17:57:09.302853 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.302855    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-4khsr\""
Jul 27 17:57:09.307320 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.307295    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"]
Jul 27 17:57:09.447455 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.447414    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37f0988-732b-4878-9875-8b95eefa828e-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kh9ks\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.447455 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.447462    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxr7d\" (UniqueName: \"kubernetes.io/projected/d37f0988-732b-4878-9875-8b95eefa828e-kube-api-access-nxr7d\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kh9ks\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.447703 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.447525    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37f0988-732b-4878-9875-8b95eefa828e-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kh9ks\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.447703 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.447561    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37f0988-732b-4878-9875-8b95eefa828e-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kh9ks\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.548614 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.548512    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxr7d\" (UniqueName: \"kubernetes.io/projected/d37f0988-732b-4878-9875-8b95eefa828e-kube-api-access-nxr7d\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kh9ks\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.548614 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.548584    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37f0988-732b-4878-9875-8b95eefa828e-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kh9ks\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.548821 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.548624    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37f0988-732b-4878-9875-8b95eefa828e-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kh9ks\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.548821 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.548693    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37f0988-732b-4878-9875-8b95eefa828e-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kh9ks\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.549064 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.549042    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37f0988-732b-4878-9875-8b95eefa828e-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kh9ks\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.549104 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.549047    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37f0988-732b-4878-9875-8b95eefa828e-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kh9ks\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.551116 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.551091    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37f0988-732b-4878-9875-8b95eefa828e-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kh9ks\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.557906 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.557880    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxr7d\" (UniqueName: \"kubernetes.io/projected/d37f0988-732b-4878-9875-8b95eefa828e-kube-api-access-nxr7d\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kh9ks\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.615168 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.615134    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:09.745965 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:09.745930    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"]
Jul 27 17:57:09.747236 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:57:09.747213    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37f0988_732b_4878_9875_8b95eefa828e.slice/crio-ebd26c958a21d85128a7cd4a34eea50f5f034210ae19a94a188983761f1a1f69 WatchSource:0}: Error finding container ebd26c958a21d85128a7cd4a34eea50f5f034210ae19a94a188983761f1a1f69: Status 404 returned error can't find the container with id ebd26c958a21d85128a7cd4a34eea50f5f034210ae19a94a188983761f1a1f69
Jul 27 17:57:10.191319 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:10.191219    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks" event={"ID":"d37f0988-732b-4878-9875-8b95eefa828e","Type":"ContainerStarted","Data":"260a8c3017884c35dd8d143ed632a7e480c0c140ac7bdaddc590f6e75dd99f42"}
Jul 27 17:57:10.191319 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:10.191278    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks" event={"ID":"d37f0988-732b-4878-9875-8b95eefa828e","Type":"ContainerStarted","Data":"ebd26c958a21d85128a7cd4a34eea50f5f034210ae19a94a188983761f1a1f69"}
Jul 27 17:57:10.191590 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:10.191351    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:10.211550 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:10.211480    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks" podStartSLOduration=1.211465372 podStartE2EDuration="1.211465372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:57:10.208814794 +0000 UTC m=+1712.253552876" watchObservedRunningTime="2026-07-27 17:57:10.211465372 +0000 UTC m=+1712.256203452"
Jul 27 17:57:20.096816 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:20.096782    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"
Jul 27 17:57:41.197962 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:41.197931    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:42.653305 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:42.653276    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"]
Jul 27 17:57:51.197438 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:51.197400    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks" podUID="d37f0988-732b-4878-9875-8b95eefa828e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:57:57.672446 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:57.672402    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks" podUID="d37f0988-732b-4878-9875-8b95eefa828e" containerName="main" containerID="cri-o://260a8c3017884c35dd8d143ed632a7e480c0c140ac7bdaddc590f6e75dd99f42" gracePeriod=45
Jul 27 17:57:57.911993 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:57.911963    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:57.924650 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:57.924597    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37f0988-732b-4878-9875-8b95eefa828e-tls-certs\") pod \"d37f0988-732b-4878-9875-8b95eefa828e\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") "
Jul 27 17:57:57.924757 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:57.924664    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37f0988-732b-4878-9875-8b95eefa828e-tokenizer-cache\") pod \"d37f0988-732b-4878-9875-8b95eefa828e\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") "
Jul 27 17:57:57.924797 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:57.924757    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37f0988-732b-4878-9875-8b95eefa828e-cachi2-cache\") pod \"d37f0988-732b-4878-9875-8b95eefa828e\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") "
Jul 27 17:57:57.924844 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:57.924818    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxr7d\" (UniqueName: \"kubernetes.io/projected/d37f0988-732b-4878-9875-8b95eefa828e-kube-api-access-nxr7d\") pod \"d37f0988-732b-4878-9875-8b95eefa828e\" (UID: \"d37f0988-732b-4878-9875-8b95eefa828e\") "
Jul 27 17:57:57.924894 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:57.924852    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37f0988-732b-4878-9875-8b95eefa828e-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d37f0988-732b-4878-9875-8b95eefa828e" (UID: "d37f0988-732b-4878-9875-8b95eefa828e"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:57:57.925023 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:57.924992    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37f0988-732b-4878-9875-8b95eefa828e-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "d37f0988-732b-4878-9875-8b95eefa828e" (UID: "d37f0988-732b-4878-9875-8b95eefa828e"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:57:57.925154 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:57.925074    2571 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37f0988-732b-4878-9875-8b95eefa828e-tokenizer-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:57:57.926807 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:57.926783    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d37f0988-732b-4878-9875-8b95eefa828e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d37f0988-732b-4878-9875-8b95eefa828e" (UID: "d37f0988-732b-4878-9875-8b95eefa828e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:57:57.926897 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:57.926809    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37f0988-732b-4878-9875-8b95eefa828e-kube-api-access-nxr7d" (OuterVolumeSpecName: "kube-api-access-nxr7d") pod "d37f0988-732b-4878-9875-8b95eefa828e" (UID: "d37f0988-732b-4878-9875-8b95eefa828e"). InnerVolumeSpecName "kube-api-access-nxr7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:57:58.025854 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:58.025821    2571 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37f0988-732b-4878-9875-8b95eefa828e-cachi2-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:57:58.025854 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:58.025851    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxr7d\" (UniqueName: \"kubernetes.io/projected/d37f0988-732b-4878-9875-8b95eefa828e-kube-api-access-nxr7d\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:57:58.025854 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:58.025861    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37f0988-732b-4878-9875-8b95eefa828e-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:57:58.421828 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:58.421796    2571 generic.go:356] "Generic (PLEG): container finished" podID="d37f0988-732b-4878-9875-8b95eefa828e" containerID="260a8c3017884c35dd8d143ed632a7e480c0c140ac7bdaddc590f6e75dd99f42" exitCode=0
Jul 27 17:57:58.421970 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:58.421838    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"
Jul 27 17:57:58.421970 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:58.421884    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks" event={"ID":"d37f0988-732b-4878-9875-8b95eefa828e","Type":"ContainerDied","Data":"260a8c3017884c35dd8d143ed632a7e480c0c140ac7bdaddc590f6e75dd99f42"}
Jul 27 17:57:58.421970 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:58.421928    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks" event={"ID":"d37f0988-732b-4878-9875-8b95eefa828e","Type":"ContainerDied","Data":"ebd26c958a21d85128a7cd4a34eea50f5f034210ae19a94a188983761f1a1f69"}
Jul 27 17:57:58.422650 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:58.421951    2571 scope.go:117] "RemoveContainer" containerID="260a8c3017884c35dd8d143ed632a7e480c0c140ac7bdaddc590f6e75dd99f42"
Jul 27 17:57:58.431615 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:58.431599    2571 scope.go:117] "RemoveContainer" containerID="260a8c3017884c35dd8d143ed632a7e480c0c140ac7bdaddc590f6e75dd99f42"
Jul 27 17:57:58.431876 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:57:58.431853    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260a8c3017884c35dd8d143ed632a7e480c0c140ac7bdaddc590f6e75dd99f42\": container with ID starting with 260a8c3017884c35dd8d143ed632a7e480c0c140ac7bdaddc590f6e75dd99f42 not found: ID does not exist" containerID="260a8c3017884c35dd8d143ed632a7e480c0c140ac7bdaddc590f6e75dd99f42"
Jul 27 17:57:58.431973 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:58.431881    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260a8c3017884c35dd8d143ed632a7e480c0c140ac7bdaddc590f6e75dd99f42"} err="failed to get container status \"260a8c3017884c35dd8d143ed632a7e480c0c140ac7bdaddc590f6e75dd99f42\": rpc error: code = NotFound desc = could not find container \"260a8c3017884c35dd8d143ed632a7e480c0c140ac7bdaddc590f6e75dd99f42\": container with ID starting with 260a8c3017884c35dd8d143ed632a7e480c0c140ac7bdaddc590f6e75dd99f42 not found: ID does not exist"
Jul 27 17:57:58.441434 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:58.441411    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"]
Jul 27 17:57:58.445078 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:57:58.445056    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kh9ks"]
Jul 27 17:58:00.399125 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:00.399082    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d37f0988-732b-4878-9875-8b95eefa828e" path="/var/lib/kubelet/pods/d37f0988-732b-4878-9875-8b95eefa828e/volumes"
Jul 27 17:58:11.648327 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.648292    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"]
Jul 27 17:58:11.648789 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.648609    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d37f0988-732b-4878-9875-8b95eefa828e" containerName="main"
Jul 27 17:58:11.648789 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.648621    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d37f0988-732b-4878-9875-8b95eefa828e" containerName="main"
Jul 27 17:58:11.648789 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.648677    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d37f0988-732b-4878-9875-8b95eefa828e" containerName="main"
Jul 27 17:58:11.651459 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.651441    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.655237 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.655208    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-qlspx\""
Jul 27 17:58:11.655378 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.655212    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 27 17:58:11.655378 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.655299    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 27 17:58:11.655378 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.655310    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 27 17:58:11.660846 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.660824    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"]
Jul 27 17:58:11.734746 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.734713    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.734913 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.734761    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.734913 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.734809    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.734913 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.734835    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x66l\" (UniqueName: \"kubernetes.io/projected/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-kube-api-access-9x66l\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.734913 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.734895    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.735077 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.734925    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.836135 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.836101    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.836135 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.836135    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x66l\" (UniqueName: \"kubernetes.io/projected/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-kube-api-access-9x66l\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.836379 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.836165    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.836379 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.836185    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.836379 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.836223    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.836379 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.836262    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.836626 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.836595    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.836626 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.836614    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.836725 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.836642    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.838606 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.838548    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.839041 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.839020    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.846220 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.846177    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x66l\" (UniqueName: \"kubernetes.io/projected/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-kube-api-access-9x66l\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:11.962594 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:11.962554    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:12.091260 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:12.091228    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"]
Jul 27 17:58:12.094557 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:58:12.094531    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36eca51d_c21c_4bf7_bbd8_84616b9e0a49.slice/crio-00fcf490aa3e67fa4703d89b13fcec21fe8030eae47d50e0bbb2927d8bceb3a8 WatchSource:0}: Error finding container 00fcf490aa3e67fa4703d89b13fcec21fe8030eae47d50e0bbb2927d8bceb3a8: Status 404 returned error can't find the container with id 00fcf490aa3e67fa4703d89b13fcec21fe8030eae47d50e0bbb2927d8bceb3a8
Jul 27 17:58:12.096361 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:12.096342    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:58:12.490154 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:12.490121    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr" event={"ID":"36eca51d-c21c-4bf7-bbd8-84616b9e0a49","Type":"ContainerStarted","Data":"00fcf490aa3e67fa4703d89b13fcec21fe8030eae47d50e0bbb2927d8bceb3a8"}
Jul 27 17:58:14.503447 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:14.503404    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr" event={"ID":"36eca51d-c21c-4bf7-bbd8-84616b9e0a49","Type":"ContainerStarted","Data":"851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9"}
Jul 27 17:58:14.519791 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:14.519745    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr" podStartSLOduration=1.970444383 podStartE2EDuration="3.519731759s" totalImagesPullingTime="1.549287376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:58:14.518402443 +0000 UTC m=+1776.563140524" watchObservedRunningTime="2026-07-27 17:58:14.519731759 +0000 UTC m=+1776.564469854"
Jul 27 17:58:14.962989 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:14.962953    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:21.963199 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:21.963158    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:21.975750 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:21.975725    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:22.552921 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:22.552893    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:45.766708 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:45.766673    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"]
Jul 27 17:58:45.775521 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:58:45.775472    2571 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:58:45.775521 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:58:45.775521 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 17:58:45.775521 ip-10-0-131-74 kubenswrapper[2571]:  > containerID="851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9" cmd=["/bin/sleep","15"]
Jul 27 17:58:45.775887 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:45.775536    2571 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr" message="" err=<
Jul 27 17:58:45.775887 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:58:45.775887 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 17:58:45.775887 ip-10-0-131-74 kubenswrapper[2571]:  >
Jul 27 17:58:45.775887 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:58:45.775548    2571 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:58:45.775887 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:58:45.775887 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 17:58:45.775887 ip-10-0-131-74 kubenswrapper[2571]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr" podUID="36eca51d-c21c-4bf7-bbd8-84616b9e0a49" containerName="main" containerID="cri-o://851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9"
Jul 27 17:58:45.775887 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:45.775610    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr" podUID="36eca51d-c21c-4bf7-bbd8-84616b9e0a49" containerName="main" containerID="cri-o://851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9" gracePeriod=60
Jul 27 17:58:46.125363 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.125337    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:46.228428 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.228393    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-home\") pod \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") "
Jul 27 17:58:46.228635 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.228463    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-model-cache\") pod \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") "
Jul 27 17:58:46.228635 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.228549    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-tls-certs\") pod \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") "
Jul 27 17:58:46.228635 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.228581    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x66l\" (UniqueName: \"kubernetes.io/projected/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-kube-api-access-9x66l\") pod \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") "
Jul 27 17:58:46.228635 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.228606    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-dshm\") pod \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") "
Jul 27 17:58:46.228868 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.228647    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-tmp-dir\") pod \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\" (UID: \"36eca51d-c21c-4bf7-bbd8-84616b9e0a49\") "
Jul 27 17:58:46.228868 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.228734    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-home" (OuterVolumeSpecName: "home") pod "36eca51d-c21c-4bf7-bbd8-84616b9e0a49" (UID: "36eca51d-c21c-4bf7-bbd8-84616b9e0a49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:58:46.228868 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.228744    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-model-cache" (OuterVolumeSpecName: "model-cache") pod "36eca51d-c21c-4bf7-bbd8-84616b9e0a49" (UID: "36eca51d-c21c-4bf7-bbd8-84616b9e0a49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:58:46.229025 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.228941    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:58:46.229025 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.228963    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:58:46.229025 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.228938    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36eca51d-c21c-4bf7-bbd8-84616b9e0a49" (UID: "36eca51d-c21c-4bf7-bbd8-84616b9e0a49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:58:46.230927 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.230903    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36eca51d-c21c-4bf7-bbd8-84616b9e0a49" (UID: "36eca51d-c21c-4bf7-bbd8-84616b9e0a49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:58:46.230927 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.230910    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-dshm" (OuterVolumeSpecName: "dshm") pod "36eca51d-c21c-4bf7-bbd8-84616b9e0a49" (UID: "36eca51d-c21c-4bf7-bbd8-84616b9e0a49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:58:46.231098 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.230924    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-kube-api-access-9x66l" (OuterVolumeSpecName: "kube-api-access-9x66l") pod "36eca51d-c21c-4bf7-bbd8-84616b9e0a49" (UID: "36eca51d-c21c-4bf7-bbd8-84616b9e0a49"). InnerVolumeSpecName "kube-api-access-9x66l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:58:46.329728 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.329697    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:58:46.329728 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.329727    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x66l\" (UniqueName: \"kubernetes.io/projected/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-kube-api-access-9x66l\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:58:46.329904 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.329737    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:58:46.329904 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.329768    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36eca51d-c21c-4bf7-bbd8-84616b9e0a49-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:58:46.659688 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.659597    2571 generic.go:356] "Generic (PLEG): container finished" podID="36eca51d-c21c-4bf7-bbd8-84616b9e0a49" containerID="851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9" exitCode=0
Jul 27 17:58:46.659688 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.659646    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr" event={"ID":"36eca51d-c21c-4bf7-bbd8-84616b9e0a49","Type":"ContainerDied","Data":"851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9"}
Jul 27 17:58:46.659688 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.659673    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"
Jul 27 17:58:46.659993 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.659702    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr" event={"ID":"36eca51d-c21c-4bf7-bbd8-84616b9e0a49","Type":"ContainerDied","Data":"00fcf490aa3e67fa4703d89b13fcec21fe8030eae47d50e0bbb2927d8bceb3a8"}
Jul 27 17:58:46.659993 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.659730    2571 scope.go:117] "RemoveContainer" containerID="851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9"
Jul 27 17:58:46.669960 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.669930    2571 scope.go:117] "RemoveContainer" containerID="851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9"
Jul 27 17:58:46.670262 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:58:46.670222    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9\": container with ID starting with 851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9 not found: ID does not exist" containerID="851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9"
Jul 27 17:58:46.670364 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.670262    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9"} err="failed to get container status \"851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9\": rpc error: code = NotFound desc = could not find container \"851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9\": container with ID starting with 851d31e865d7e920030c2170f24047db23acb56c2c3e48027346718514b778e9 not found: ID does not exist"
Jul 27 17:58:46.676391 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.676369    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"]
Jul 27 17:58:46.683824 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:46.683801    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-mvdrr"]
Jul 27 17:58:48.398610 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:58:48.398576    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36eca51d-c21c-4bf7-bbd8-84616b9e0a49" path="/var/lib/kubelet/pods/36eca51d-c21c-4bf7-bbd8-84616b9e0a49/volumes"
Jul 27 17:59:20.625268 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.625232    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"]
Jul 27 17:59:20.625893 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.625709    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36eca51d-c21c-4bf7-bbd8-84616b9e0a49" containerName="main"
Jul 27 17:59:20.625893 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.625729    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="36eca51d-c21c-4bf7-bbd8-84616b9e0a49" containerName="main"
Jul 27 17:59:20.625893 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.625815    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="36eca51d-c21c-4bf7-bbd8-84616b9e0a49" containerName="main"
Jul 27 17:59:20.628801 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.628780    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.633063 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.633033    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-zz7t6\""
Jul 27 17:59:20.633063 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.633051    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 27 17:59:20.633255 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.633054    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 27 17:59:20.633255 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.633061    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 27 17:59:20.639006 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.638985    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"]
Jul 27 17:59:20.710272 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.710236    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k7c4\" (UniqueName: \"kubernetes.io/projected/1caa3bcb-05ed-495f-9341-3d086aa7640d-kube-api-access-7k7c4\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.710272 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.710275    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.710515 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.710296    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1caa3bcb-05ed-495f-9341-3d086aa7640d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.710515 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.710339    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.710515 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.710370    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.710515 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.710405    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.811646 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.811613    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k7c4\" (UniqueName: \"kubernetes.io/projected/1caa3bcb-05ed-495f-9341-3d086aa7640d-kube-api-access-7k7c4\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.811834 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.811655    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.811834 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.811674    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1caa3bcb-05ed-495f-9341-3d086aa7640d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.811834 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.811704    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.811834 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.811742    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.811834 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.811788    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.812190 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.812167    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.812314 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.812196    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.812314 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.812245    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.814093 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.814071    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.814208 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.814191    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1caa3bcb-05ed-495f-9341-3d086aa7640d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.822403 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.822376    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k7c4\" (UniqueName: \"kubernetes.io/projected/1caa3bcb-05ed-495f-9341-3d086aa7640d-kube-api-access-7k7c4\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:20.940595 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:20.940483    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:21.070729 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:21.070700    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"]
Jul 27 17:59:21.072607 ip-10-0-131-74 kubenswrapper[2571]: W0727 17:59:21.072574    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caa3bcb_05ed_495f_9341_3d086aa7640d.slice/crio-b7e375c859bad933d68b2389c2a379c9177d751aabf83e91cd21bdf5141990f1 WatchSource:0}: Error finding container b7e375c859bad933d68b2389c2a379c9177d751aabf83e91cd21bdf5141990f1: Status 404 returned error can't find the container with id b7e375c859bad933d68b2389c2a379c9177d751aabf83e91cd21bdf5141990f1
Jul 27 17:59:21.832374 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:21.832333    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf" event={"ID":"1caa3bcb-05ed-495f-9341-3d086aa7640d","Type":"ContainerStarted","Data":"9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c"}
Jul 27 17:59:21.832851 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:21.832438    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf" event={"ID":"1caa3bcb-05ed-495f-9341-3d086aa7640d","Type":"ContainerStarted","Data":"b7e375c859bad933d68b2389c2a379c9177d751aabf83e91cd21bdf5141990f1"}
Jul 27 17:59:21.850757 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:21.850693    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf" podStartSLOduration=1.850671755 podStartE2EDuration="1.850671755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:59:21.848149368 +0000 UTC m=+1843.892887451" watchObservedRunningTime="2026-07-27 17:59:21.850671755 +0000 UTC m=+1843.895409837"
Jul 27 17:59:21.941367 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:21.941321    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:30.941187 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:30.941148    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:30.953987 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:30.953955    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:31.893360 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:31.893325    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:54.203533 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.203480    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"]
Jul 27 17:59:54.212816 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:59:54.212782    2571 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:59:54.212816 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:59:54.212816 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 17:59:54.212816 ip-10-0-131-74 kubenswrapper[2571]:  > containerID="9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c" cmd=["/bin/sleep","15"]
Jul 27 17:59:54.213129 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.212833    2571 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf" message="" err=<
Jul 27 17:59:54.213129 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:59:54.213129 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 17:59:54.213129 ip-10-0-131-74 kubenswrapper[2571]:  >
Jul 27 17:59:54.213129 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:59:54.212849    2571 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:59:54.213129 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:59:54.213129 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 17:59:54.213129 ip-10-0-131-74 kubenswrapper[2571]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf" podUID="1caa3bcb-05ed-495f-9341-3d086aa7640d" containerName="main" containerID="cri-o://9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c"
Jul 27 17:59:54.213129 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.212891    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf" podUID="1caa3bcb-05ed-495f-9341-3d086aa7640d" containerName="main" containerID="cri-o://9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c" gracePeriod=60
Jul 27 17:59:54.553319 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.553293    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:54.704846 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.704803    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-home\") pod \"1caa3bcb-05ed-495f-9341-3d086aa7640d\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") "
Jul 27 17:59:54.705056 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.704871    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1caa3bcb-05ed-495f-9341-3d086aa7640d-tls-certs\") pod \"1caa3bcb-05ed-495f-9341-3d086aa7640d\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") "
Jul 27 17:59:54.705056 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.704911    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-model-cache\") pod \"1caa3bcb-05ed-495f-9341-3d086aa7640d\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") "
Jul 27 17:59:54.705056 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.704926    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-tmp-dir\") pod \"1caa3bcb-05ed-495f-9341-3d086aa7640d\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") "
Jul 27 17:59:54.705056 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.704944    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-dshm\") pod \"1caa3bcb-05ed-495f-9341-3d086aa7640d\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") "
Jul 27 17:59:54.705056 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.704968    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k7c4\" (UniqueName: \"kubernetes.io/projected/1caa3bcb-05ed-495f-9341-3d086aa7640d-kube-api-access-7k7c4\") pod \"1caa3bcb-05ed-495f-9341-3d086aa7640d\" (UID: \"1caa3bcb-05ed-495f-9341-3d086aa7640d\") "
Jul 27 17:59:54.705289 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.705144    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-home" (OuterVolumeSpecName: "home") pod "1caa3bcb-05ed-495f-9341-3d086aa7640d" (UID: "1caa3bcb-05ed-495f-9341-3d086aa7640d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:59:54.705289 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.705206    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1caa3bcb-05ed-495f-9341-3d086aa7640d" (UID: "1caa3bcb-05ed-495f-9341-3d086aa7640d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:59:54.705289 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.705213    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1caa3bcb-05ed-495f-9341-3d086aa7640d" (UID: "1caa3bcb-05ed-495f-9341-3d086aa7640d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:59:54.707177 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.707145    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-dshm" (OuterVolumeSpecName: "dshm") pod "1caa3bcb-05ed-495f-9341-3d086aa7640d" (UID: "1caa3bcb-05ed-495f-9341-3d086aa7640d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:59:54.707291 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.707198    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1caa3bcb-05ed-495f-9341-3d086aa7640d-kube-api-access-7k7c4" (OuterVolumeSpecName: "kube-api-access-7k7c4") pod "1caa3bcb-05ed-495f-9341-3d086aa7640d" (UID: "1caa3bcb-05ed-495f-9341-3d086aa7640d"). InnerVolumeSpecName "kube-api-access-7k7c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:59:54.707291 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.707259    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1caa3bcb-05ed-495f-9341-3d086aa7640d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1caa3bcb-05ed-495f-9341-3d086aa7640d" (UID: "1caa3bcb-05ed-495f-9341-3d086aa7640d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:59:54.805915 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.805826    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:59:54.805915 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.805866    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:59:54.805915 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.805875    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:59:54.805915 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.805883    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k7c4\" (UniqueName: \"kubernetes.io/projected/1caa3bcb-05ed-495f-9341-3d086aa7640d-kube-api-access-7k7c4\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:59:54.805915 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.805893    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1caa3bcb-05ed-495f-9341-3d086aa7640d-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:59:54.805915 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:54.805902    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1caa3bcb-05ed-495f-9341-3d086aa7640d-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 17:59:55.002603 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:55.002564    2571 generic.go:356] "Generic (PLEG): container finished" podID="1caa3bcb-05ed-495f-9341-3d086aa7640d" containerID="9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c" exitCode=0
Jul 27 17:59:55.002805 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:55.002633    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"
Jul 27 17:59:55.002805 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:55.002651    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf" event={"ID":"1caa3bcb-05ed-495f-9341-3d086aa7640d","Type":"ContainerDied","Data":"9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c"}
Jul 27 17:59:55.002805 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:55.002734    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf" event={"ID":"1caa3bcb-05ed-495f-9341-3d086aa7640d","Type":"ContainerDied","Data":"b7e375c859bad933d68b2389c2a379c9177d751aabf83e91cd21bdf5141990f1"}
Jul 27 17:59:55.002805 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:55.002756    2571 scope.go:117] "RemoveContainer" containerID="9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c"
Jul 27 17:59:55.012399 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:55.012382    2571 scope.go:117] "RemoveContainer" containerID="9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c"
Jul 27 17:59:55.012663 ip-10-0-131-74 kubenswrapper[2571]: E0727 17:59:55.012644    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c\": container with ID starting with 9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c not found: ID does not exist" containerID="9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c"
Jul 27 17:59:55.012731 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:55.012671    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c"} err="failed to get container status \"9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c\": rpc error: code = NotFound desc = could not find container \"9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c\": container with ID starting with 9df0f924ab191d50e96387f3e49e73d9d2c3f5c0a4ccb7fb7b12c072d178c84c not found: ID does not exist"
Jul 27 17:59:55.024171 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:55.024142    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"]
Jul 27 17:59:55.027979 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:55.027949    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkn6kf"]
Jul 27 17:59:56.398317 ip-10-0-131-74 kubenswrapper[2571]: I0727 17:59:56.398284    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1caa3bcb-05ed-495f-9341-3d086aa7640d" path="/var/lib/kubelet/pods/1caa3bcb-05ed-495f-9341-3d086aa7640d/volumes"
Jul 27 18:00:25.023826 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.023781    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"]
Jul 27 18:00:25.024421 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.024142    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1caa3bcb-05ed-495f-9341-3d086aa7640d" containerName="main"
Jul 27 18:00:25.024421 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.024155    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1caa3bcb-05ed-495f-9341-3d086aa7640d" containerName="main"
Jul 27 18:00:25.024421 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.024215    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1caa3bcb-05ed-495f-9341-3d086aa7640d" containerName="main"
Jul 27 18:00:25.027207 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.027183    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.029860 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.029841    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 27 18:00:25.030914 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.030880    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 27 18:00:25.031043 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.030946    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 27 18:00:25.031043 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.030882    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 27 18:00:25.031043 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.031029    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-rvs2g\""
Jul 27 18:00:25.038098 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.038075    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"]
Jul 27 18:00:25.156009 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.155962    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.156009 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.156013    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-home\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.156248 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.156042    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqx9b\" (UniqueName: \"kubernetes.io/projected/380a5a1d-b187-48e9-b8b2-d2883c8f768f-kube-api-access-hqx9b\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.156248 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.156098    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-model-cache\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.156248 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.156135    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.156248 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.156199    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/380a5a1d-b187-48e9-b8b2-d2883c8f768f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.156248 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.156241    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-dshm\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.257227 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.257186    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.257227 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.257230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-home\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.257527 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.257267    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqx9b\" (UniqueName: \"kubernetes.io/projected/380a5a1d-b187-48e9-b8b2-d2883c8f768f-kube-api-access-hqx9b\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.257527 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.257298    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-model-cache\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.257527 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.257322    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.257527 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.257358    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/380a5a1d-b187-48e9-b8b2-d2883c8f768f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.257527 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.257385    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-dshm\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.257821 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.257699    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.257821 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.257715    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-model-cache\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.257821 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.257775    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-home\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.257821 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.257800    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.259679 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.259650    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-dshm\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.259831 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.259815    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/380a5a1d-b187-48e9-b8b2-d2883c8f768f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.265588 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.265568    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqx9b\" (UniqueName: \"kubernetes.io/projected/380a5a1d-b187-48e9-b8b2-d2883c8f768f-kube-api-access-hqx9b\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.338016 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.337925    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:25.483996 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:25.483956    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"]
Jul 27 18:00:25.487326 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:00:25.487296    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380a5a1d_b187_48e9_b8b2_d2883c8f768f.slice/crio-487c8b739989503af8a9c32d9483b682573f36d94b3701f506288b1d4c10709d WatchSource:0}: Error finding container 487c8b739989503af8a9c32d9483b682573f36d94b3701f506288b1d4c10709d: Status 404 returned error can't find the container with id 487c8b739989503af8a9c32d9483b682573f36d94b3701f506288b1d4c10709d
Jul 27 18:00:26.164139 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:26.164086    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c" event={"ID":"380a5a1d-b187-48e9-b8b2-d2883c8f768f","Type":"ContainerStarted","Data":"749d32a265994728c71a8519737fde7fdc619522643114e147674b5c52606550"}
Jul 27 18:00:26.164692 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:26.164162    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c" event={"ID":"380a5a1d-b187-48e9-b8b2-d2883c8f768f","Type":"ContainerStarted","Data":"487c8b739989503af8a9c32d9483b682573f36d94b3701f506288b1d4c10709d"}
Jul 27 18:00:32.199227 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:32.199140    2571 generic.go:356] "Generic (PLEG): container finished" podID="380a5a1d-b187-48e9-b8b2-d2883c8f768f" containerID="749d32a265994728c71a8519737fde7fdc619522643114e147674b5c52606550" exitCode=0
Jul 27 18:00:32.199637 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:32.199219    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c" event={"ID":"380a5a1d-b187-48e9-b8b2-d2883c8f768f","Type":"ContainerDied","Data":"749d32a265994728c71a8519737fde7fdc619522643114e147674b5c52606550"}
Jul 27 18:00:33.205770 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:33.205731    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c" event={"ID":"380a5a1d-b187-48e9-b8b2-d2883c8f768f","Type":"ContainerStarted","Data":"708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e"}
Jul 27 18:00:33.227452 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:33.227392    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c" podStartSLOduration=3.023787572 podStartE2EDuration="9.227377849s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.203590277s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:00:33.226566949 +0000 UTC m=+1915.271305029" watchObservedRunningTime="2026-07-27 18:00:33.227377849 +0000 UTC m=+1915.272115930"
Jul 27 18:00:33.338240 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:33.338203    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:35.338639 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:35.338603    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:35.351330 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:35.351296    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:00:36.231567 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:00:36.231537    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:02:25.149814 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.149778    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"]
Jul 27 18:02:25.159254 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:02:25.159222    2571 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 18:02:25.159254 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:02:25.159254 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:02:25.159254 ip-10-0-131-74 kubenswrapper[2571]:  > containerID="708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e" cmd=["/bin/sleep","15"]
Jul 27 18:02:25.159572 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.159270    2571 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c" message="" err=<
Jul 27 18:02:25.159572 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:02:25.159572 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:02:25.159572 ip-10-0-131-74 kubenswrapper[2571]:  >
Jul 27 18:02:25.159572 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:02:25.159282    2571 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 18:02:25.159572 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:02:25.159572 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:02:25.159572 ip-10-0-131-74 kubenswrapper[2571]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c" podUID="380a5a1d-b187-48e9-b8b2-d2883c8f768f" containerName="main" containerID="cri-o://708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e"
Jul 27 18:02:25.159572 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.159322    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c" podUID="380a5a1d-b187-48e9-b8b2-d2883c8f768f" containerName="main" containerID="cri-o://708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e" gracePeriod=60
Jul 27 18:02:25.221413 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.221377    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c" podUID="380a5a1d-b187-48e9-b8b2-d2883c8f768f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 27 18:02:25.513669 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.513645    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:02:25.599034 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.598996    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-kserve-provision-location\") pod \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") "
Jul 27 18:02:25.599213 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.599054    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-tmp-dir\") pod \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") "
Jul 27 18:02:25.599213 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.599077    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-dshm\") pod \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") "
Jul 27 18:02:25.599213 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.599106    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqx9b\" (UniqueName: \"kubernetes.io/projected/380a5a1d-b187-48e9-b8b2-d2883c8f768f-kube-api-access-hqx9b\") pod \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") "
Jul 27 18:02:25.599213 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.599121    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/380a5a1d-b187-48e9-b8b2-d2883c8f768f-tls-certs\") pod \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") "
Jul 27 18:02:25.599213 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.599156    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-model-cache\") pod \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") "
Jul 27 18:02:25.599213 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.599210    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-home\") pod \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\" (UID: \"380a5a1d-b187-48e9-b8b2-d2883c8f768f\") "
Jul 27 18:02:25.599562 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.599424    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "380a5a1d-b187-48e9-b8b2-d2883c8f768f" (UID: "380a5a1d-b187-48e9-b8b2-d2883c8f768f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:02:25.599631 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.599577    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-model-cache" (OuterVolumeSpecName: "model-cache") pod "380a5a1d-b187-48e9-b8b2-d2883c8f768f" (UID: "380a5a1d-b187-48e9-b8b2-d2883c8f768f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:02:25.599683 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.599625    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-home" (OuterVolumeSpecName: "home") pod "380a5a1d-b187-48e9-b8b2-d2883c8f768f" (UID: "380a5a1d-b187-48e9-b8b2-d2883c8f768f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:02:25.601911 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.601880    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-dshm" (OuterVolumeSpecName: "dshm") pod "380a5a1d-b187-48e9-b8b2-d2883c8f768f" (UID: "380a5a1d-b187-48e9-b8b2-d2883c8f768f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:02:25.602018 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.601951    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380a5a1d-b187-48e9-b8b2-d2883c8f768f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "380a5a1d-b187-48e9-b8b2-d2883c8f768f" (UID: "380a5a1d-b187-48e9-b8b2-d2883c8f768f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:02:25.602018 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.601970    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380a5a1d-b187-48e9-b8b2-d2883c8f768f-kube-api-access-hqx9b" (OuterVolumeSpecName: "kube-api-access-hqx9b") pod "380a5a1d-b187-48e9-b8b2-d2883c8f768f" (UID: "380a5a1d-b187-48e9-b8b2-d2883c8f768f"). InnerVolumeSpecName "kube-api-access-hqx9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:02:25.663608 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.663467    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "380a5a1d-b187-48e9-b8b2-d2883c8f768f" (UID: "380a5a1d-b187-48e9-b8b2-d2883c8f768f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:02:25.700782 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.700690    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.700782 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.700725    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.700782 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.700735    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqx9b\" (UniqueName: \"kubernetes.io/projected/380a5a1d-b187-48e9-b8b2-d2883c8f768f-kube-api-access-hqx9b\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.700782 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.700744    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/380a5a1d-b187-48e9-b8b2-d2883c8f768f-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.700782 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.700753    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.700782 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.700761    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.700782 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.700768    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380a5a1d-b187-48e9-b8b2-d2883c8f768f-kserve-provision-location\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.752320 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.752287    2571 generic.go:356] "Generic (PLEG): container finished" podID="380a5a1d-b187-48e9-b8b2-d2883c8f768f" containerID="708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e" exitCode=0
Jul 27 18:02:25.752537 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.752335    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"
Jul 27 18:02:25.752537 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.752370    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c" event={"ID":"380a5a1d-b187-48e9-b8b2-d2883c8f768f","Type":"ContainerDied","Data":"708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e"}
Jul 27 18:02:25.752537 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.752434    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c" event={"ID":"380a5a1d-b187-48e9-b8b2-d2883c8f768f","Type":"ContainerDied","Data":"487c8b739989503af8a9c32d9483b682573f36d94b3701f506288b1d4c10709d"}
Jul 27 18:02:25.752537 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.752465    2571 scope.go:117] "RemoveContainer" containerID="708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e"
Jul 27 18:02:25.762208 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.762193    2571 scope.go:117] "RemoveContainer" containerID="749d32a265994728c71a8519737fde7fdc619522643114e147674b5c52606550"
Jul 27 18:02:25.770387 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.770370    2571 scope.go:117] "RemoveContainer" containerID="708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e"
Jul 27 18:02:25.770652 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:02:25.770634    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e\": container with ID starting with 708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e not found: ID does not exist" containerID="708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e"
Jul 27 18:02:25.770711 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.770661    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e"} err="failed to get container status \"708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e\": rpc error: code = NotFound desc = could not find container \"708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e\": container with ID starting with 708e3a81e9039ba7866bb29491ef7ec17c6a7b2fcfc4c0072b4593e61d368f8e not found: ID does not exist"
Jul 27 18:02:25.770711 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.770679    2571 scope.go:117] "RemoveContainer" containerID="749d32a265994728c71a8519737fde7fdc619522643114e147674b5c52606550"
Jul 27 18:02:25.770904 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:02:25.770891    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749d32a265994728c71a8519737fde7fdc619522643114e147674b5c52606550\": container with ID starting with 749d32a265994728c71a8519737fde7fdc619522643114e147674b5c52606550 not found: ID does not exist" containerID="749d32a265994728c71a8519737fde7fdc619522643114e147674b5c52606550"
Jul 27 18:02:25.770943 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.770908    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749d32a265994728c71a8519737fde7fdc619522643114e147674b5c52606550"} err="failed to get container status \"749d32a265994728c71a8519737fde7fdc619522643114e147674b5c52606550\": rpc error: code = NotFound desc = could not find container \"749d32a265994728c71a8519737fde7fdc619522643114e147674b5c52606550\": container with ID starting with 749d32a265994728c71a8519737fde7fdc619522643114e147674b5c52606550 not found: ID does not exist"
Jul 27 18:02:25.776934 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.776907    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"]
Jul 27 18:02:25.783205 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:25.783182    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-j4r2c"]
Jul 27 18:02:26.398473 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:02:26.398428    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="380a5a1d-b187-48e9-b8b2-d2883c8f768f" path="/var/lib/kubelet/pods/380a5a1d-b187-48e9-b8b2-d2883c8f768f/volumes"
Jul 27 18:11:29.753906 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:29.753867    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"]
Jul 27 18:11:30.097378 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:30.097289    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m" podUID="23131789-0408-45db-969c-7ddcdeba9d07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:11:40.096819 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:40.096774    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m" podUID="23131789-0408-45db-969c-7ddcdeba9d07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:11:44.772943 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:44.772893    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m" podUID="23131789-0408-45db-969c-7ddcdeba9d07" containerName="main" containerID="cri-o://ff345e52cf0519adc9232526f6313339772cab72efdbf6d9e66d6097c490d4f2" gracePeriod=45
Jul 27 18:11:45.028809 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.028750    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"
Jul 27 18:11:45.109686 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.109654    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd8l4\" (UniqueName: \"kubernetes.io/projected/23131789-0408-45db-969c-7ddcdeba9d07-kube-api-access-zd8l4\") pod \"23131789-0408-45db-969c-7ddcdeba9d07\" (UID: \"23131789-0408-45db-969c-7ddcdeba9d07\") "
Jul 27 18:11:45.109873 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.109691    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23131789-0408-45db-969c-7ddcdeba9d07-tls-certs\") pod \"23131789-0408-45db-969c-7ddcdeba9d07\" (UID: \"23131789-0408-45db-969c-7ddcdeba9d07\") "
Jul 27 18:11:45.111921 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.111877    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23131789-0408-45db-969c-7ddcdeba9d07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23131789-0408-45db-969c-7ddcdeba9d07" (UID: "23131789-0408-45db-969c-7ddcdeba9d07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:11:45.111921 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.111879    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23131789-0408-45db-969c-7ddcdeba9d07-kube-api-access-zd8l4" (OuterVolumeSpecName: "kube-api-access-zd8l4") pod "23131789-0408-45db-969c-7ddcdeba9d07" (UID: "23131789-0408-45db-969c-7ddcdeba9d07"). InnerVolumeSpecName "kube-api-access-zd8l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:11:45.210809 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.210769    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd8l4\" (UniqueName: \"kubernetes.io/projected/23131789-0408-45db-969c-7ddcdeba9d07-kube-api-access-zd8l4\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:11:45.210809 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.210803    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23131789-0408-45db-969c-7ddcdeba9d07-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:11:45.474470 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.474438    2571 generic.go:356] "Generic (PLEG): container finished" podID="23131789-0408-45db-969c-7ddcdeba9d07" containerID="ff345e52cf0519adc9232526f6313339772cab72efdbf6d9e66d6097c490d4f2" exitCode=0
Jul 27 18:11:45.474672 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.474482    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"
Jul 27 18:11:45.474672 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.474527    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m" event={"ID":"23131789-0408-45db-969c-7ddcdeba9d07","Type":"ContainerDied","Data":"ff345e52cf0519adc9232526f6313339772cab72efdbf6d9e66d6097c490d4f2"}
Jul 27 18:11:45.474672 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.474585    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m" event={"ID":"23131789-0408-45db-969c-7ddcdeba9d07","Type":"ContainerDied","Data":"353f515b59171f8093f5537f3670a745b74f47a6527ac18e823d15c4739839f5"}
Jul 27 18:11:45.474672 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.474609    2571 scope.go:117] "RemoveContainer" containerID="ff345e52cf0519adc9232526f6313339772cab72efdbf6d9e66d6097c490d4f2"
Jul 27 18:11:45.483527 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.483488    2571 scope.go:117] "RemoveContainer" containerID="ff345e52cf0519adc9232526f6313339772cab72efdbf6d9e66d6097c490d4f2"
Jul 27 18:11:45.483774 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:11:45.483755    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff345e52cf0519adc9232526f6313339772cab72efdbf6d9e66d6097c490d4f2\": container with ID starting with ff345e52cf0519adc9232526f6313339772cab72efdbf6d9e66d6097c490d4f2 not found: ID does not exist" containerID="ff345e52cf0519adc9232526f6313339772cab72efdbf6d9e66d6097c490d4f2"
Jul 27 18:11:45.483842 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.483787    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff345e52cf0519adc9232526f6313339772cab72efdbf6d9e66d6097c490d4f2"} err="failed to get container status \"ff345e52cf0519adc9232526f6313339772cab72efdbf6d9e66d6097c490d4f2\": rpc error: code = NotFound desc = could not find container \"ff345e52cf0519adc9232526f6313339772cab72efdbf6d9e66d6097c490d4f2\": container with ID starting with ff345e52cf0519adc9232526f6313339772cab72efdbf6d9e66d6097c490d4f2 not found: ID does not exist"
Jul 27 18:11:45.498022 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.497995    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"]
Jul 27 18:11:45.503018 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:45.502995    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m"]
Jul 27 18:11:46.399205 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:11:46.399170    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23131789-0408-45db-969c-7ddcdeba9d07" path="/var/lib/kubelet/pods/23131789-0408-45db-969c-7ddcdeba9d07/volumes"
Jul 27 18:12:00.785482 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.785395    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"]
Jul 27 18:12:00.785888 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.785723    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23131789-0408-45db-969c-7ddcdeba9d07" containerName="main"
Jul 27 18:12:00.785888 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.785735    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="23131789-0408-45db-969c-7ddcdeba9d07" containerName="main"
Jul 27 18:12:00.785888 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.785751    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380a5a1d-b187-48e9-b8b2-d2883c8f768f" containerName="main"
Jul 27 18:12:00.785888 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.785758    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="380a5a1d-b187-48e9-b8b2-d2883c8f768f" containerName="main"
Jul 27 18:12:00.785888 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.785765    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380a5a1d-b187-48e9-b8b2-d2883c8f768f" containerName="storage-initializer"
Jul 27 18:12:00.785888 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.785770    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="380a5a1d-b187-48e9-b8b2-d2883c8f768f" containerName="storage-initializer"
Jul 27 18:12:00.785888 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.785822    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="23131789-0408-45db-969c-7ddcdeba9d07" containerName="main"
Jul 27 18:12:00.785888 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.785829    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="380a5a1d-b187-48e9-b8b2-d2883c8f768f" containerName="main"
Jul 27 18:12:00.788855 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.788839    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.791736 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.791715    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"kube-root-ca.crt\""
Jul 27 18:12:00.792877 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.792856    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"default-dockercfg-4jwxt\""
Jul 27 18:12:00.792993 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.792884    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-kserve-self-signed-certs\""
Jul 27 18:12:00.792993 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.792887    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"openshift-service-ca.crt\""
Jul 27 18:12:00.798315 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.798293    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"]
Jul 27 18:12:00.846612 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.846575    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.846792 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.846626    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.846792 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.846656    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65452329-e447-4dc1-8119-f124c30e0246-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.846792 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.846696    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.846792 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.846789    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j67h\" (UniqueName: \"kubernetes.io/projected/65452329-e447-4dc1-8119-f124c30e0246-kube-api-access-6j67h\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.846940 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.846809    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.948047 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.948014    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j67h\" (UniqueName: \"kubernetes.io/projected/65452329-e447-4dc1-8119-f124c30e0246-kube-api-access-6j67h\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.948047 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.948050    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.948344 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.948225    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.948344 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.948282    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.948344 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.948317    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65452329-e447-4dc1-8119-f124c30e0246-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.948344 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.948341    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.948585 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.948436    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.948656 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.948629    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.948717 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.948697    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.950576 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.950486    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.950707 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.950648    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65452329-e447-4dc1-8119-f124c30e0246-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:00.956925 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:00.956899    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j67h\" (UniqueName: \"kubernetes.io/projected/65452329-e447-4dc1-8119-f124c30e0246-kube-api-access-6j67h\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:01.100178 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:01.100082    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:01.231374 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:01.231346    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"]
Jul 27 18:12:01.233660 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:12:01.233633    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65452329_e447_4dc1_8119_f124c30e0246.slice/crio-772c5751ec4278ce4ae1996fb7d558d82c3c2fe428216a31cd1486e4275dce01 WatchSource:0}: Error finding container 772c5751ec4278ce4ae1996fb7d558d82c3c2fe428216a31cd1486e4275dce01: Status 404 returned error can't find the container with id 772c5751ec4278ce4ae1996fb7d558d82c3c2fe428216a31cd1486e4275dce01
Jul 27 18:12:01.553078 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:01.553043    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l" event={"ID":"65452329-e447-4dc1-8119-f124c30e0246","Type":"ContainerStarted","Data":"84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2"}
Jul 27 18:12:01.553275 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:01.553105    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l" event={"ID":"65452329-e447-4dc1-8119-f124c30e0246","Type":"ContainerStarted","Data":"772c5751ec4278ce4ae1996fb7d558d82c3c2fe428216a31cd1486e4275dce01"}
Jul 27 18:12:01.571348 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:01.571289    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l" podStartSLOduration=1.5712736550000002 podStartE2EDuration="1.571273655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:12:01.570547142 +0000 UTC m=+2603.615285227" watchObservedRunningTime="2026-07-27 18:12:01.571273655 +0000 UTC m=+2603.616011736"
Jul 27 18:12:02.100976 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:02.100920    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:11.100763 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:11.100725    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:11.113385 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:11.113354    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:12:11.612948 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:12:11.612918    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:18:41.619706 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.619668    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"]
Jul 27 18:18:41.622935 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.622917    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.626474 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.626453    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 27 18:18:41.626474 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.626469    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 27 18:18:41.627424 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.627405    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 27 18:18:41.627592 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.627570    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-44gpj\""
Jul 27 18:18:41.627657 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.627570    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 27 18:18:41.646461 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.646436    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"]
Jul 27 18:18:41.742805 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.742771    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-home\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.742805 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.742807    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpnpd\" (UniqueName: \"kubernetes.io/projected/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-kube-api-access-dpnpd\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.743075 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.742832    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-model-cache\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.743075 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.742904    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.743075 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.742947    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.743075 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.742983    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.743075 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.743023    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-dshm\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.844472 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.844432    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-home\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.844472 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.844475    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpnpd\" (UniqueName: \"kubernetes.io/projected/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-kube-api-access-dpnpd\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.844759 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.844537    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-model-cache\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.844759 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.844566    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.844759 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.844592    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.844759 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.844623    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.844759 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.844669    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-dshm\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.844998 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.844941    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-home\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.844998 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.844982    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.845074 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.845000    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-model-cache\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.845074 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.845062    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.846915 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.846885    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-dshm\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.847233 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.847211    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.854006 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.853987    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpnpd\" (UniqueName: \"kubernetes.io/projected/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-kube-api-access-dpnpd\") pod \"tokenizer-migration-path-test-kserve-569c688787-vlcrw\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.932888 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.932790    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:18:41.946738 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.946708    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"]
Jul 27 18:18:41.950412 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.950388    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"
Jul 27 18:18:41.956146 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.956119    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-jchxb\""
Jul 27 18:18:41.966850 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:41.966824    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"]
Jul 27 18:18:42.046424 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:42.046387    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrl6l\" (UniqueName: \"kubernetes.io/projected/ab818e6d-cfff-4627-af86-1242c211e6c5-kube-api-access-wrl6l\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx\" (UID: \"ab818e6d-cfff-4627-af86-1242c211e6c5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"
Jul 27 18:18:42.046639 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:42.046467    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab818e6d-cfff-4627-af86-1242c211e6c5-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx\" (UID: \"ab818e6d-cfff-4627-af86-1242c211e6c5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"
Jul 27 18:18:42.092271 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:42.087348    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"]
Jul 27 18:18:42.093859 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:18:42.093826    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df609ce_ec8c_4dfd_8ccb_e93582c16dd9.slice/crio-05b33f7df040e5a14d5e3fc2de7d9de97ec3ee9bfc7d1e3b920f737cd8e7ec3f WatchSource:0}: Error finding container 05b33f7df040e5a14d5e3fc2de7d9de97ec3ee9bfc7d1e3b920f737cd8e7ec3f: Status 404 returned error can't find the container with id 05b33f7df040e5a14d5e3fc2de7d9de97ec3ee9bfc7d1e3b920f737cd8e7ec3f
Jul 27 18:18:42.147151 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:42.147126    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab818e6d-cfff-4627-af86-1242c211e6c5-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx\" (UID: \"ab818e6d-cfff-4627-af86-1242c211e6c5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"
Jul 27 18:18:42.147270 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:42.147196    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrl6l\" (UniqueName: \"kubernetes.io/projected/ab818e6d-cfff-4627-af86-1242c211e6c5-kube-api-access-wrl6l\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx\" (UID: \"ab818e6d-cfff-4627-af86-1242c211e6c5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"
Jul 27 18:18:42.149488 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:42.149468    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab818e6d-cfff-4627-af86-1242c211e6c5-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx\" (UID: \"ab818e6d-cfff-4627-af86-1242c211e6c5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"
Jul 27 18:18:42.157871 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:42.157853    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrl6l\" (UniqueName: \"kubernetes.io/projected/ab818e6d-cfff-4627-af86-1242c211e6c5-kube-api-access-wrl6l\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx\" (UID: \"ab818e6d-cfff-4627-af86-1242c211e6c5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"
Jul 27 18:18:42.266643 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:42.266603    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"
Jul 27 18:18:42.412774 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:42.412742    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"]
Jul 27 18:18:42.413473 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:18:42.413439    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab818e6d_cfff_4627_af86_1242c211e6c5.slice/crio-e4707f12d6184002d1d5d04dfe91aceadc003fe3415359d02efd91c1df75c4aa WatchSource:0}: Error finding container e4707f12d6184002d1d5d04dfe91aceadc003fe3415359d02efd91c1df75c4aa: Status 404 returned error can't find the container with id e4707f12d6184002d1d5d04dfe91aceadc003fe3415359d02efd91c1df75c4aa
Jul 27 18:18:42.469137 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:42.469106    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx" event={"ID":"ab818e6d-cfff-4627-af86-1242c211e6c5","Type":"ContainerStarted","Data":"e4707f12d6184002d1d5d04dfe91aceadc003fe3415359d02efd91c1df75c4aa"}
Jul 27 18:18:42.470725 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:42.470697    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw" event={"ID":"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9","Type":"ContainerStarted","Data":"e66c9c6d997203423ccbb09540863437c21d8e88566b337bb550e139a4737dbb"}
Jul 27 18:18:42.470823 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:42.470749    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw" event={"ID":"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9","Type":"ContainerStarted","Data":"05b33f7df040e5a14d5e3fc2de7d9de97ec3ee9bfc7d1e3b920f737cd8e7ec3f"}
Jul 27 18:18:43.480051 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:43.479944    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx" event={"ID":"ab818e6d-cfff-4627-af86-1242c211e6c5","Type":"ContainerStarted","Data":"52e2a2d6133674e706d04cc5d66ca5961232bb633939149fa6b7179446af31d0"}
Jul 27 18:18:43.480517 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:43.480086    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"
Jul 27 18:18:43.498259 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:43.498207    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx" podStartSLOduration=2.498189887 podStartE2EDuration="2.498189887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:18:43.497532111 +0000 UTC m=+3005.542270195" watchObservedRunningTime="2026-07-27 18:18:43.498189887 +0000 UTC m=+3005.542928034"
Jul 27 18:18:51.522711 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:51.522674    2571 generic.go:356] "Generic (PLEG): container finished" podID="3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" containerID="e66c9c6d997203423ccbb09540863437c21d8e88566b337bb550e139a4737dbb" exitCode=0
Jul 27 18:18:51.523300 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:51.522750    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw" event={"ID":"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9","Type":"ContainerDied","Data":"e66c9c6d997203423ccbb09540863437c21d8e88566b337bb550e139a4737dbb"}
Jul 27 18:18:52.529837 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:52.529801    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw" event={"ID":"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9","Type":"ContainerStarted","Data":"be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182"}
Jul 27 18:18:52.549917 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:52.549863    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw" podStartSLOduration=2.389048612 podStartE2EDuration="11.549848982s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.16080037s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:18:52.54794609 +0000 UTC m=+3014.592684212" watchObservedRunningTime="2026-07-27 18:18:52.549848982 +0000 UTC m=+3014.594587063"
Jul 27 18:18:52.933177 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:18:52.933094    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:19:01.933307 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:01.933270    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:19:01.945905 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:01.945880    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:19:02.588601 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:02.588567    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:19:14.486727 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:14.486696    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"
Jul 27 18:19:27.477456 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.477419    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"]
Jul 27 18:19:27.488857 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.488827    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"]
Jul 27 18:19:27.498006 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:19:27.497966    2571 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 18:19:27.498006 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:19:27.498006 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:19:27.498006 ip-10-0-131-74 kubenswrapper[2571]:  > containerID="be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182" cmd=["/bin/sleep","15"]
Jul 27 18:19:27.498302 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.498020    2571 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw" message="" err=<
Jul 27 18:19:27.498302 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:19:27.498302 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:19:27.498302 ip-10-0-131-74 kubenswrapper[2571]:  >
Jul 27 18:19:27.498302 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:19:27.498032    2571 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 18:19:27.498302 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:19:27.498302 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:19:27.498302 ip-10-0-131-74 kubenswrapper[2571]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw" podUID="3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" containerName="main" containerID="cri-o://be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182"
Jul 27 18:19:27.498302 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.498077    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw" podUID="3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" containerName="main" containerID="cri-o://be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182" gracePeriod=60
Jul 27 18:19:27.577632 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.577590    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw" podUID="3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 27 18:19:27.859213 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.859183    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:19:27.911128 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.911024    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-home\") pod \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") "
Jul 27 18:19:27.911308 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.911195    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-kserve-provision-location\") pod \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") "
Jul 27 18:19:27.911308 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.911231    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-dshm\") pod \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") "
Jul 27 18:19:27.911308 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.911259    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-home" (OuterVolumeSpecName: "home") pod "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" (UID: "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:19:27.911308 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.911266    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-model-cache\") pod \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") "
Jul 27 18:19:27.911566 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.911336    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpnpd\" (UniqueName: \"kubernetes.io/projected/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-kube-api-access-dpnpd\") pod \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") "
Jul 27 18:19:27.911566 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.911358    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-tls-certs\") pod \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") "
Jul 27 18:19:27.911566 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.911380    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-tmp-dir\") pod \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\" (UID: \"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9\") "
Jul 27 18:19:27.911566 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.911489    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-model-cache" (OuterVolumeSpecName: "model-cache") pod "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" (UID: "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:19:27.911750 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.911638    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" (UID: "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:19:27.911816 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.911795    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:19:27.911866 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.911830    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:19:27.911866 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.911841    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:19:27.913742 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.913698    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" (UID: "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:19:27.913742 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.913732    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-dshm" (OuterVolumeSpecName: "dshm") pod "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" (UID: "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:19:27.913924 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.913647    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-kube-api-access-dpnpd" (OuterVolumeSpecName: "kube-api-access-dpnpd") pod "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" (UID: "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9"). InnerVolumeSpecName "kube-api-access-dpnpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:19:27.973380 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:27.973326    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" (UID: "3df609ce-ec8c-4dfd-8ccb-e93582c16dd9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:19:28.012313 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.012274    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-kserve-provision-location\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:19:28.012313 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.012309    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:19:28.012561 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.012325    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpnpd\" (UniqueName: \"kubernetes.io/projected/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-kube-api-access-dpnpd\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:19:28.012561 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.012335    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:19:28.707556 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.707528    2571 generic.go:356] "Generic (PLEG): container finished" podID="3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" containerID="be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182" exitCode=0
Jul 27 18:19:28.708086 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.707569    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"
Jul 27 18:19:28.708086 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.707605    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw" event={"ID":"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9","Type":"ContainerDied","Data":"be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182"}
Jul 27 18:19:28.708086 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.707678    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw" event={"ID":"3df609ce-ec8c-4dfd-8ccb-e93582c16dd9","Type":"ContainerDied","Data":"05b33f7df040e5a14d5e3fc2de7d9de97ec3ee9bfc7d1e3b920f737cd8e7ec3f"}
Jul 27 18:19:28.708086 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.707719    2571 scope.go:117] "RemoveContainer" containerID="be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182"
Jul 27 18:19:28.716771 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.716754    2571 scope.go:117] "RemoveContainer" containerID="e66c9c6d997203423ccbb09540863437c21d8e88566b337bb550e139a4737dbb"
Jul 27 18:19:28.724846 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.724823    2571 scope.go:117] "RemoveContainer" containerID="be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182"
Jul 27 18:19:28.725102 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:19:28.725076    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182\": container with ID starting with be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182 not found: ID does not exist" containerID="be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182"
Jul 27 18:19:28.725159 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.725112    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182"} err="failed to get container status \"be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182\": rpc error: code = NotFound desc = could not find container \"be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182\": container with ID starting with be831200eefa300f621741961784c7e39cf889cf1406899d222643631f435182 not found: ID does not exist"
Jul 27 18:19:28.725159 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.725131    2571 scope.go:117] "RemoveContainer" containerID="e66c9c6d997203423ccbb09540863437c21d8e88566b337bb550e139a4737dbb"
Jul 27 18:19:28.725358 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:19:28.725339    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66c9c6d997203423ccbb09540863437c21d8e88566b337bb550e139a4737dbb\": container with ID starting with e66c9c6d997203423ccbb09540863437c21d8e88566b337bb550e139a4737dbb not found: ID does not exist" containerID="e66c9c6d997203423ccbb09540863437c21d8e88566b337bb550e139a4737dbb"
Jul 27 18:19:28.725417 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.725366    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66c9c6d997203423ccbb09540863437c21d8e88566b337bb550e139a4737dbb"} err="failed to get container status \"e66c9c6d997203423ccbb09540863437c21d8e88566b337bb550e139a4737dbb\": rpc error: code = NotFound desc = could not find container \"e66c9c6d997203423ccbb09540863437c21d8e88566b337bb550e139a4737dbb\": container with ID starting with e66c9c6d997203423ccbb09540863437c21d8e88566b337bb550e139a4737dbb not found: ID does not exist"
Jul 27 18:19:28.728927 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.728903    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"]
Jul 27 18:19:28.733352 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:28.733330    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-vlcrw"]
Jul 27 18:19:30.399056 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:30.399011    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" path="/var/lib/kubelet/pods/3df609ce-ec8c-4dfd-8ccb-e93582c16dd9/volumes"
Jul 27 18:19:34.486848 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:34.486805    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx" podUID="ab818e6d-cfff-4627-af86-1242c211e6c5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:19:42.498403 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.498331    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx" podUID="ab818e6d-cfff-4627-af86-1242c211e6c5" containerName="main" containerID="cri-o://52e2a2d6133674e706d04cc5d66ca5961232bb633939149fa6b7179446af31d0" gracePeriod=45
Jul 27 18:19:42.756815 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.756743    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"
Jul 27 18:19:42.776783 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.776739    2571 generic.go:356] "Generic (PLEG): container finished" podID="ab818e6d-cfff-4627-af86-1242c211e6c5" containerID="52e2a2d6133674e706d04cc5d66ca5961232bb633939149fa6b7179446af31d0" exitCode=0
Jul 27 18:19:42.776968 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.776792    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx" event={"ID":"ab818e6d-cfff-4627-af86-1242c211e6c5","Type":"ContainerDied","Data":"52e2a2d6133674e706d04cc5d66ca5961232bb633939149fa6b7179446af31d0"}
Jul 27 18:19:42.776968 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.776852    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"
Jul 27 18:19:42.776968 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.776869    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx" event={"ID":"ab818e6d-cfff-4627-af86-1242c211e6c5","Type":"ContainerDied","Data":"e4707f12d6184002d1d5d04dfe91aceadc003fe3415359d02efd91c1df75c4aa"}
Jul 27 18:19:42.776968 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.776904    2571 scope.go:117] "RemoveContainer" containerID="52e2a2d6133674e706d04cc5d66ca5961232bb633939149fa6b7179446af31d0"
Jul 27 18:19:42.785890 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.785869    2571 scope.go:117] "RemoveContainer" containerID="52e2a2d6133674e706d04cc5d66ca5961232bb633939149fa6b7179446af31d0"
Jul 27 18:19:42.786161 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:19:42.786141    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e2a2d6133674e706d04cc5d66ca5961232bb633939149fa6b7179446af31d0\": container with ID starting with 52e2a2d6133674e706d04cc5d66ca5961232bb633939149fa6b7179446af31d0 not found: ID does not exist" containerID="52e2a2d6133674e706d04cc5d66ca5961232bb633939149fa6b7179446af31d0"
Jul 27 18:19:42.786228 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.786170    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e2a2d6133674e706d04cc5d66ca5961232bb633939149fa6b7179446af31d0"} err="failed to get container status \"52e2a2d6133674e706d04cc5d66ca5961232bb633939149fa6b7179446af31d0\": rpc error: code = NotFound desc = could not find container \"52e2a2d6133674e706d04cc5d66ca5961232bb633939149fa6b7179446af31d0\": container with ID starting with 52e2a2d6133674e706d04cc5d66ca5961232bb633939149fa6b7179446af31d0 not found: ID does not exist"
Jul 27 18:19:42.834646 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.834610    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab818e6d-cfff-4627-af86-1242c211e6c5-tls-certs\") pod \"ab818e6d-cfff-4627-af86-1242c211e6c5\" (UID: \"ab818e6d-cfff-4627-af86-1242c211e6c5\") "
Jul 27 18:19:42.834814 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.834700    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrl6l\" (UniqueName: \"kubernetes.io/projected/ab818e6d-cfff-4627-af86-1242c211e6c5-kube-api-access-wrl6l\") pod \"ab818e6d-cfff-4627-af86-1242c211e6c5\" (UID: \"ab818e6d-cfff-4627-af86-1242c211e6c5\") "
Jul 27 18:19:42.836813 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.836779    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab818e6d-cfff-4627-af86-1242c211e6c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab818e6d-cfff-4627-af86-1242c211e6c5" (UID: "ab818e6d-cfff-4627-af86-1242c211e6c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:19:42.836813 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.836785    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab818e6d-cfff-4627-af86-1242c211e6c5-kube-api-access-wrl6l" (OuterVolumeSpecName: "kube-api-access-wrl6l") pod "ab818e6d-cfff-4627-af86-1242c211e6c5" (UID: "ab818e6d-cfff-4627-af86-1242c211e6c5"). InnerVolumeSpecName "kube-api-access-wrl6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:19:42.936140 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.936100    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrl6l\" (UniqueName: \"kubernetes.io/projected/ab818e6d-cfff-4627-af86-1242c211e6c5-kube-api-access-wrl6l\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:19:42.936140 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:42.936131    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab818e6d-cfff-4627-af86-1242c211e6c5-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:19:43.099478 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:43.099447    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"]
Jul 27 18:19:43.103198 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:43.103169    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ppklx"]
Jul 27 18:19:44.398476 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:44.398432    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab818e6d-cfff-4627-af86-1242c211e6c5" path="/var/lib/kubelet/pods/ab818e6d-cfff-4627-af86-1242c211e6c5/volumes"
Jul 27 18:19:53.912098 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.912062    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"]
Jul 27 18:19:53.912553 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.912398    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" containerName="main"
Jul 27 18:19:53.912553 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.912411    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" containerName="main"
Jul 27 18:19:53.912553 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.912434    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab818e6d-cfff-4627-af86-1242c211e6c5" containerName="main"
Jul 27 18:19:53.912553 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.912441    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab818e6d-cfff-4627-af86-1242c211e6c5" containerName="main"
Jul 27 18:19:53.912553 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.912450    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" containerName="storage-initializer"
Jul 27 18:19:53.912553 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.912456    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" containerName="storage-initializer"
Jul 27 18:19:53.912553 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.912525    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab818e6d-cfff-4627-af86-1242c211e6c5" containerName="main"
Jul 27 18:19:53.912553 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.912532    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3df609ce-ec8c-4dfd-8ccb-e93582c16dd9" containerName="main"
Jul 27 18:19:53.915465 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.915448    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"
Jul 27 18:19:53.918291 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.918265    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 27 18:19:53.919564 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.919544    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 27 18:19:53.919645 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.919580    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 27 18:19:53.919690 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.919662    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-4t6g8\""
Jul 27 18:19:53.919808 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.919794    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-r6ctb\""
Jul 27 18:19:53.924374 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:53.924350    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"]
Jul 27 18:19:54.032692 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:54.032658    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx459\" (UniqueName: \"kubernetes.io/projected/f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4-kube-api-access-hx459\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6\" (UID: \"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"
Jul 27 18:19:54.032874 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:54.032696    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6\" (UID: \"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"
Jul 27 18:19:54.133399 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:54.133365    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx459\" (UniqueName: \"kubernetes.io/projected/f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4-kube-api-access-hx459\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6\" (UID: \"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"
Jul 27 18:19:54.133399 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:54.133400    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6\" (UID: \"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"
Jul 27 18:19:54.135958 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:54.135916    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6\" (UID: \"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"
Jul 27 18:19:54.144107 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:54.144078    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx459\" (UniqueName: \"kubernetes.io/projected/f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4-kube-api-access-hx459\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6\" (UID: \"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"
Jul 27 18:19:54.226597 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:54.226566    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"
Jul 27 18:19:54.361347 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:54.361315    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"]
Jul 27 18:19:54.365008 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:19:54.364980    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b22fb3_b8b6_42fb_85ad_1e0e63ba3ee4.slice/crio-041bff9608e187519bbe4ce100de23167d62166eac5b5e7df549adea465860e1 WatchSource:0}: Error finding container 041bff9608e187519bbe4ce100de23167d62166eac5b5e7df549adea465860e1: Status 404 returned error can't find the container with id 041bff9608e187519bbe4ce100de23167d62166eac5b5e7df549adea465860e1
Jul 27 18:19:54.841820 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:54.841781    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6" event={"ID":"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4","Type":"ContainerStarted","Data":"0462af63295fb67b82caa28af433c9fca404e3a40f0b01d97e273be53a48795c"}
Jul 27 18:19:54.842004 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:54.841833    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6" event={"ID":"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4","Type":"ContainerStarted","Data":"041bff9608e187519bbe4ce100de23167d62166eac5b5e7df549adea465860e1"}
Jul 27 18:19:54.842004 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:54.841868    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"
Jul 27 18:19:54.862091 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:19:54.862038    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6" podStartSLOduration=1.8620238059999998 podStartE2EDuration="1.862023806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:19:54.859760154 +0000 UTC m=+3076.904498234" watchObservedRunningTime="2026-07-27 18:19:54.862023806 +0000 UTC m=+3076.906761886"
Jul 27 18:20:25.848436 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:25.848401    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"
Jul 27 18:20:27.244560 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:27.244515    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"]
Jul 27 18:20:35.848604 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:35.848565    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6" podUID="f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:20:42.263914 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:42.263840    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6" podUID="f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4" containerName="main" containerID="cri-o://0462af63295fb67b82caa28af433c9fca404e3a40f0b01d97e273be53a48795c" gracePeriod=45
Jul 27 18:20:42.519026 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:42.518964    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"
Jul 27 18:20:42.670684 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:42.670640    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4-tls-certs\") pod \"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4\" (UID: \"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4\") "
Jul 27 18:20:42.670879 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:42.670753    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx459\" (UniqueName: \"kubernetes.io/projected/f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4-kube-api-access-hx459\") pod \"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4\" (UID: \"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4\") "
Jul 27 18:20:42.672941 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:42.672909    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4" (UID: "f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:20:42.673060 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:42.673017    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4-kube-api-access-hx459" (OuterVolumeSpecName: "kube-api-access-hx459") pod "f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4" (UID: "f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4"). InnerVolumeSpecName "kube-api-access-hx459". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:20:42.771722 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:42.771630    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx459\" (UniqueName: \"kubernetes.io/projected/f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4-kube-api-access-hx459\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:20:42.771722 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:42.771666    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:20:43.085671 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:43.085580    2571 generic.go:356] "Generic (PLEG): container finished" podID="f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4" containerID="0462af63295fb67b82caa28af433c9fca404e3a40f0b01d97e273be53a48795c" exitCode=0
Jul 27 18:20:43.085671 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:43.085622    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6" event={"ID":"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4","Type":"ContainerDied","Data":"0462af63295fb67b82caa28af433c9fca404e3a40f0b01d97e273be53a48795c"}
Jul 27 18:20:43.085671 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:43.085627    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"
Jul 27 18:20:43.085918 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:43.085672    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6" event={"ID":"f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4","Type":"ContainerDied","Data":"041bff9608e187519bbe4ce100de23167d62166eac5b5e7df549adea465860e1"}
Jul 27 18:20:43.085918 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:43.085697    2571 scope.go:117] "RemoveContainer" containerID="0462af63295fb67b82caa28af433c9fca404e3a40f0b01d97e273be53a48795c"
Jul 27 18:20:43.095526 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:43.095482    2571 scope.go:117] "RemoveContainer" containerID="0462af63295fb67b82caa28af433c9fca404e3a40f0b01d97e273be53a48795c"
Jul 27 18:20:43.095805 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:20:43.095783    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0462af63295fb67b82caa28af433c9fca404e3a40f0b01d97e273be53a48795c\": container with ID starting with 0462af63295fb67b82caa28af433c9fca404e3a40f0b01d97e273be53a48795c not found: ID does not exist" containerID="0462af63295fb67b82caa28af433c9fca404e3a40f0b01d97e273be53a48795c"
Jul 27 18:20:43.095861 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:43.095814    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0462af63295fb67b82caa28af433c9fca404e3a40f0b01d97e273be53a48795c"} err="failed to get container status \"0462af63295fb67b82caa28af433c9fca404e3a40f0b01d97e273be53a48795c\": rpc error: code = NotFound desc = could not find container \"0462af63295fb67b82caa28af433c9fca404e3a40f0b01d97e273be53a48795c\": container with ID starting with 0462af63295fb67b82caa28af433c9fca404e3a40f0b01d97e273be53a48795c not found: ID does not exist"
Jul 27 18:20:43.107066 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:43.107043    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"]
Jul 27 18:20:43.113019 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:43.112994    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebx9q6"]
Jul 27 18:20:44.399685 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:44.399644    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4" path="/var/lib/kubelet/pods/f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4/volumes"
Jul 27 18:20:53.243103 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.243064    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"]
Jul 27 18:20:53.243601 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.243395    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4" containerName="main"
Jul 27 18:20:53.243601 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.243408    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4" containerName="main"
Jul 27 18:20:53.243601 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.243478    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9b22fb3-b8b6-42fb-85ad-1e0e63ba3ee4" containerName="main"
Jul 27 18:20:53.246403 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.246385    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"
Jul 27 18:20:53.250439 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.250413    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 27 18:20:53.250439 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.250416    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 27 18:20:53.250662 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.250465    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 27 18:20:53.250662 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.250467    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-n86fh\""
Jul 27 18:20:53.250662 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.250509    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-nrm2c\""
Jul 27 18:20:53.253926 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.253901    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsbb7\" (UniqueName: \"kubernetes.io/projected/c798efe4-741f-43b3-a3cf-30d944d40f98-kube-api-access-bsbb7\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf\" (UID: \"c798efe4-741f-43b3-a3cf-30d944d40f98\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"
Jul 27 18:20:53.254051 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.253985    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c798efe4-741f-43b3-a3cf-30d944d40f98-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf\" (UID: \"c798efe4-741f-43b3-a3cf-30d944d40f98\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"
Jul 27 18:20:53.255356 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.255330    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"]
Jul 27 18:20:53.354383 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.354346    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsbb7\" (UniqueName: \"kubernetes.io/projected/c798efe4-741f-43b3-a3cf-30d944d40f98-kube-api-access-bsbb7\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf\" (UID: \"c798efe4-741f-43b3-a3cf-30d944d40f98\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"
Jul 27 18:20:53.354597 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.354431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c798efe4-741f-43b3-a3cf-30d944d40f98-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf\" (UID: \"c798efe4-741f-43b3-a3cf-30d944d40f98\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"
Jul 27 18:20:53.357042 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.357009    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c798efe4-741f-43b3-a3cf-30d944d40f98-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf\" (UID: \"c798efe4-741f-43b3-a3cf-30d944d40f98\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"
Jul 27 18:20:53.367886 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.367860    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsbb7\" (UniqueName: \"kubernetes.io/projected/c798efe4-741f-43b3-a3cf-30d944d40f98-kube-api-access-bsbb7\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf\" (UID: \"c798efe4-741f-43b3-a3cf-30d944d40f98\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"
Jul 27 18:20:53.558106 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.558009    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"
Jul 27 18:20:53.692766 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:53.692712    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"]
Jul 27 18:20:53.695048 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:20:53.695015    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc798efe4_741f_43b3_a3cf_30d944d40f98.slice/crio-a501805ad20a8966028d88eed108302a56ab224245e25d7da01e03ba11ce4175 WatchSource:0}: Error finding container a501805ad20a8966028d88eed108302a56ab224245e25d7da01e03ba11ce4175: Status 404 returned error can't find the container with id a501805ad20a8966028d88eed108302a56ab224245e25d7da01e03ba11ce4175
Jul 27 18:20:54.149302 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:54.149266    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf" event={"ID":"c798efe4-741f-43b3-a3cf-30d944d40f98","Type":"ContainerStarted","Data":"e53678357778a59ac4e926323da94b92199a9dacd64ad28a9895a3b80a6de65c"}
Jul 27 18:20:54.149302 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:54.149320    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf" event={"ID":"c798efe4-741f-43b3-a3cf-30d944d40f98","Type":"ContainerStarted","Data":"a501805ad20a8966028d88eed108302a56ab224245e25d7da01e03ba11ce4175"}
Jul 27 18:20:54.149623 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:54.149440    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"
Jul 27 18:20:54.165311 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:20:54.165250    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf" podStartSLOduration=1.1652346040000001 podStartE2EDuration="1.165234604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:20:54.164305756 +0000 UTC m=+3136.209043838" watchObservedRunningTime="2026-07-27 18:20:54.165234604 +0000 UTC m=+3136.209972685"
Jul 27 18:21:25.156035 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:25.155998    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"
Jul 27 18:21:31.291621 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:31.291580    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"]
Jul 27 18:21:31.294988 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:31.294968    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"
Jul 27 18:21:31.298851 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:31.298824    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 27 18:21:31.298991 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:31.298849    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-r7djk\""
Jul 27 18:21:31.304062 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:31.304034    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"]
Jul 27 18:21:31.382004 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:31.381969    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cj9t\" (UniqueName: \"kubernetes.io/projected/25c3e4b0-1e9a-4bd8-9352-402fd36bae73-kube-api-access-5cj9t\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg\" (UID: \"25c3e4b0-1e9a-4bd8-9352-402fd36bae73\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"
Jul 27 18:21:31.382195 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:31.382116    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c3e4b0-1e9a-4bd8-9352-402fd36bae73-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg\" (UID: \"25c3e4b0-1e9a-4bd8-9352-402fd36bae73\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"
Jul 27 18:21:31.482817 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:31.482772    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c3e4b0-1e9a-4bd8-9352-402fd36bae73-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg\" (UID: \"25c3e4b0-1e9a-4bd8-9352-402fd36bae73\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"
Jul 27 18:21:31.483042 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:31.482915    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cj9t\" (UniqueName: \"kubernetes.io/projected/25c3e4b0-1e9a-4bd8-9352-402fd36bae73-kube-api-access-5cj9t\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg\" (UID: \"25c3e4b0-1e9a-4bd8-9352-402fd36bae73\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"
Jul 27 18:21:31.485390 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:31.485367    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c3e4b0-1e9a-4bd8-9352-402fd36bae73-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg\" (UID: \"25c3e4b0-1e9a-4bd8-9352-402fd36bae73\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"
Jul 27 18:21:31.490948 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:31.490875    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cj9t\" (UniqueName: \"kubernetes.io/projected/25c3e4b0-1e9a-4bd8-9352-402fd36bae73-kube-api-access-5cj9t\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg\" (UID: \"25c3e4b0-1e9a-4bd8-9352-402fd36bae73\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"
Jul 27 18:21:31.606077 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:31.605988    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"
Jul 27 18:21:31.740032 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:31.739996    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"]
Jul 27 18:21:31.743675 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:21:31.743641    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c3e4b0_1e9a_4bd8_9352_402fd36bae73.slice/crio-a93e60e42298d2a9cc32415533a843d9f29461a18d80881a7de01bb8c808cb12 WatchSource:0}: Error finding container a93e60e42298d2a9cc32415533a843d9f29461a18d80881a7de01bb8c808cb12: Status 404 returned error can't find the container with id a93e60e42298d2a9cc32415533a843d9f29461a18d80881a7de01bb8c808cb12
Jul 27 18:21:32.348457 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:32.348420    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg" event={"ID":"25c3e4b0-1e9a-4bd8-9352-402fd36bae73","Type":"ContainerStarted","Data":"d55dff56df970d5a58f0d1b1d96181eb75ba0e697897a512e723e3313fde28b4"}
Jul 27 18:21:32.348923 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:32.348471    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg" event={"ID":"25c3e4b0-1e9a-4bd8-9352-402fd36bae73","Type":"ContainerStarted","Data":"a93e60e42298d2a9cc32415533a843d9f29461a18d80881a7de01bb8c808cb12"}
Jul 27 18:21:32.348923 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:32.348516    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"
Jul 27 18:21:32.364905 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:21:32.364852    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg" podStartSLOduration=1.364834564 podStartE2EDuration="1.364834564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:21:32.363579735 +0000 UTC m=+3174.408317847" watchObservedRunningTime="2026-07-27 18:21:32.364834564 +0000 UTC m=+3174.409572644"
Jul 27 18:22:03.355768 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:03.355739    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"
Jul 27 18:22:04.527281 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:04.527238    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"]
Jul 27 18:22:13.356485 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:13.356449    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg" podUID="25c3e4b0-1e9a-4bd8-9352-402fd36bae73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:22:19.549314 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:19.549239    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg" podUID="25c3e4b0-1e9a-4bd8-9352-402fd36bae73" containerName="main" containerID="cri-o://d55dff56df970d5a58f0d1b1d96181eb75ba0e697897a512e723e3313fde28b4" gracePeriod=45
Jul 27 18:22:19.801794 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:19.801733    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"
Jul 27 18:22:19.880465 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:19.880428    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c3e4b0-1e9a-4bd8-9352-402fd36bae73-tls-certs\") pod \"25c3e4b0-1e9a-4bd8-9352-402fd36bae73\" (UID: \"25c3e4b0-1e9a-4bd8-9352-402fd36bae73\") "
Jul 27 18:22:19.880655 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:19.880542    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cj9t\" (UniqueName: \"kubernetes.io/projected/25c3e4b0-1e9a-4bd8-9352-402fd36bae73-kube-api-access-5cj9t\") pod \"25c3e4b0-1e9a-4bd8-9352-402fd36bae73\" (UID: \"25c3e4b0-1e9a-4bd8-9352-402fd36bae73\") "
Jul 27 18:22:19.882703 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:19.882672    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c3e4b0-1e9a-4bd8-9352-402fd36bae73-kube-api-access-5cj9t" (OuterVolumeSpecName: "kube-api-access-5cj9t") pod "25c3e4b0-1e9a-4bd8-9352-402fd36bae73" (UID: "25c3e4b0-1e9a-4bd8-9352-402fd36bae73"). InnerVolumeSpecName "kube-api-access-5cj9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:22:19.882832 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:19.882686    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25c3e4b0-1e9a-4bd8-9352-402fd36bae73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25c3e4b0-1e9a-4bd8-9352-402fd36bae73" (UID: "25c3e4b0-1e9a-4bd8-9352-402fd36bae73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:22:19.981485 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:19.981450    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cj9t\" (UniqueName: \"kubernetes.io/projected/25c3e4b0-1e9a-4bd8-9352-402fd36bae73-kube-api-access-5cj9t\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:22:19.981485 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:19.981479    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c3e4b0-1e9a-4bd8-9352-402fd36bae73-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:22:20.588526 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:20.588476    2571 generic.go:356] "Generic (PLEG): container finished" podID="25c3e4b0-1e9a-4bd8-9352-402fd36bae73" containerID="d55dff56df970d5a58f0d1b1d96181eb75ba0e697897a512e723e3313fde28b4" exitCode=0
Jul 27 18:22:20.588971 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:20.588537    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"
Jul 27 18:22:20.588971 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:20.588532    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg" event={"ID":"25c3e4b0-1e9a-4bd8-9352-402fd36bae73","Type":"ContainerDied","Data":"d55dff56df970d5a58f0d1b1d96181eb75ba0e697897a512e723e3313fde28b4"}
Jul 27 18:22:20.588971 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:20.588663    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg" event={"ID":"25c3e4b0-1e9a-4bd8-9352-402fd36bae73","Type":"ContainerDied","Data":"a93e60e42298d2a9cc32415533a843d9f29461a18d80881a7de01bb8c808cb12"}
Jul 27 18:22:20.588971 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:20.588686    2571 scope.go:117] "RemoveContainer" containerID="d55dff56df970d5a58f0d1b1d96181eb75ba0e697897a512e723e3313fde28b4"
Jul 27 18:22:20.597061 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:20.597041    2571 scope.go:117] "RemoveContainer" containerID="d55dff56df970d5a58f0d1b1d96181eb75ba0e697897a512e723e3313fde28b4"
Jul 27 18:22:20.597332 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:22:20.597313    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55dff56df970d5a58f0d1b1d96181eb75ba0e697897a512e723e3313fde28b4\": container with ID starting with d55dff56df970d5a58f0d1b1d96181eb75ba0e697897a512e723e3313fde28b4 not found: ID does not exist" containerID="d55dff56df970d5a58f0d1b1d96181eb75ba0e697897a512e723e3313fde28b4"
Jul 27 18:22:20.597383 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:20.597340    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55dff56df970d5a58f0d1b1d96181eb75ba0e697897a512e723e3313fde28b4"} err="failed to get container status \"d55dff56df970d5a58f0d1b1d96181eb75ba0e697897a512e723e3313fde28b4\": rpc error: code = NotFound desc = could not find container \"d55dff56df970d5a58f0d1b1d96181eb75ba0e697897a512e723e3313fde28b4\": container with ID starting with d55dff56df970d5a58f0d1b1d96181eb75ba0e697897a512e723e3313fde28b4 not found: ID does not exist"
Jul 27 18:22:20.607685 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:20.607658    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"]
Jul 27 18:22:20.613840 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:20.613817    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche46mmg"]
Jul 27 18:22:22.399028 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:22.398991    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25c3e4b0-1e9a-4bd8-9352-402fd36bae73" path="/var/lib/kubelet/pods/25c3e4b0-1e9a-4bd8-9352-402fd36bae73/volumes"
Jul 27 18:22:24.622448 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:24.622418    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"]
Jul 27 18:22:25.156641 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:25.156600    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf" podUID="c798efe4-741f-43b3-a3cf-30d944d40f98" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:22:35.155908 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:35.155819    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf" podUID="c798efe4-741f-43b3-a3cf-30d944d40f98" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:22:39.642395 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:39.642326    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf" podUID="c798efe4-741f-43b3-a3cf-30d944d40f98" containerName="main" containerID="cri-o://e53678357778a59ac4e926323da94b92199a9dacd64ad28a9895a3b80a6de65c" gracePeriod=45
Jul 27 18:22:39.889927 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:39.889902    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"
Jul 27 18:22:40.050630 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.050591    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsbb7\" (UniqueName: \"kubernetes.io/projected/c798efe4-741f-43b3-a3cf-30d944d40f98-kube-api-access-bsbb7\") pod \"c798efe4-741f-43b3-a3cf-30d944d40f98\" (UID: \"c798efe4-741f-43b3-a3cf-30d944d40f98\") "
Jul 27 18:22:40.050825 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.050651    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c798efe4-741f-43b3-a3cf-30d944d40f98-tls-certs\") pod \"c798efe4-741f-43b3-a3cf-30d944d40f98\" (UID: \"c798efe4-741f-43b3-a3cf-30d944d40f98\") "
Jul 27 18:22:40.052786 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.052755    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c798efe4-741f-43b3-a3cf-30d944d40f98-kube-api-access-bsbb7" (OuterVolumeSpecName: "kube-api-access-bsbb7") pod "c798efe4-741f-43b3-a3cf-30d944d40f98" (UID: "c798efe4-741f-43b3-a3cf-30d944d40f98"). InnerVolumeSpecName "kube-api-access-bsbb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:22:40.052907 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.052851    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c798efe4-741f-43b3-a3cf-30d944d40f98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c798efe4-741f-43b3-a3cf-30d944d40f98" (UID: "c798efe4-741f-43b3-a3cf-30d944d40f98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:22:40.151467 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.151428    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c798efe4-741f-43b3-a3cf-30d944d40f98-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:22:40.151467 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.151461    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsbb7\" (UniqueName: \"kubernetes.io/projected/c798efe4-741f-43b3-a3cf-30d944d40f98-kube-api-access-bsbb7\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:22:40.688820 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.688731    2571 generic.go:356] "Generic (PLEG): container finished" podID="c798efe4-741f-43b3-a3cf-30d944d40f98" containerID="e53678357778a59ac4e926323da94b92199a9dacd64ad28a9895a3b80a6de65c" exitCode=0
Jul 27 18:22:40.688820 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.688782    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"
Jul 27 18:22:40.689324 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.688814    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf" event={"ID":"c798efe4-741f-43b3-a3cf-30d944d40f98","Type":"ContainerDied","Data":"e53678357778a59ac4e926323da94b92199a9dacd64ad28a9895a3b80a6de65c"}
Jul 27 18:22:40.689324 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.688867    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf" event={"ID":"c798efe4-741f-43b3-a3cf-30d944d40f98","Type":"ContainerDied","Data":"a501805ad20a8966028d88eed108302a56ab224245e25d7da01e03ba11ce4175"}
Jul 27 18:22:40.689324 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.688889    2571 scope.go:117] "RemoveContainer" containerID="e53678357778a59ac4e926323da94b92199a9dacd64ad28a9895a3b80a6de65c"
Jul 27 18:22:40.697610 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.697589    2571 scope.go:117] "RemoveContainer" containerID="e53678357778a59ac4e926323da94b92199a9dacd64ad28a9895a3b80a6de65c"
Jul 27 18:22:40.697891 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:22:40.697874    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e53678357778a59ac4e926323da94b92199a9dacd64ad28a9895a3b80a6de65c\": container with ID starting with e53678357778a59ac4e926323da94b92199a9dacd64ad28a9895a3b80a6de65c not found: ID does not exist" containerID="e53678357778a59ac4e926323da94b92199a9dacd64ad28a9895a3b80a6de65c"
Jul 27 18:22:40.697952 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.697899    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e53678357778a59ac4e926323da94b92199a9dacd64ad28a9895a3b80a6de65c"} err="failed to get container status \"e53678357778a59ac4e926323da94b92199a9dacd64ad28a9895a3b80a6de65c\": rpc error: code = NotFound desc = could not find container \"e53678357778a59ac4e926323da94b92199a9dacd64ad28a9895a3b80a6de65c\": container with ID starting with e53678357778a59ac4e926323da94b92199a9dacd64ad28a9895a3b80a6de65c not found: ID does not exist"
Jul 27 18:22:40.705408 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.705382    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"]
Jul 27 18:22:40.714506 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:40.714464    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheptdtf"]
Jul 27 18:22:42.399052 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:42.399012    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c798efe4-741f-43b3-a3cf-30d944d40f98" path="/var/lib/kubelet/pods/c798efe4-741f-43b3-a3cf-30d944d40f98/volumes"
Jul 27 18:22:52.154383 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.154337    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"]
Jul 27 18:22:52.154892 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.154657    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c798efe4-741f-43b3-a3cf-30d944d40f98" containerName="main"
Jul 27 18:22:52.154892 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.154669    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c798efe4-741f-43b3-a3cf-30d944d40f98" containerName="main"
Jul 27 18:22:52.154892 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.154695    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25c3e4b0-1e9a-4bd8-9352-402fd36bae73" containerName="main"
Jul 27 18:22:52.154892 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.154701    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="25c3e4b0-1e9a-4bd8-9352-402fd36bae73" containerName="main"
Jul 27 18:22:52.154892 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.154754    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c798efe4-741f-43b3-a3cf-30d944d40f98" containerName="main"
Jul 27 18:22:52.154892 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.154767    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="25c3e4b0-1e9a-4bd8-9352-402fd36bae73" containerName="main"
Jul 27 18:22:52.158812 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.158786    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"
Jul 27 18:22:52.163357 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.163193    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 27 18:22:52.163357 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.163210    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 27 18:22:52.163622 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.163443    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 27 18:22:52.163622 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.163528    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-wgvq6\""
Jul 27 18:22:52.163622 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.163588    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-r7d8l\""
Jul 27 18:22:52.169142 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.169119    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"]
Jul 27 18:22:52.260280 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.260247    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ea779a-ccc0-44be-9f23-283f1c210221-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6\" (UID: \"b5ea779a-ccc0-44be-9f23-283f1c210221\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"
Jul 27 18:22:52.260477 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.260303    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8g5s\" (UniqueName: \"kubernetes.io/projected/b5ea779a-ccc0-44be-9f23-283f1c210221-kube-api-access-s8g5s\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6\" (UID: \"b5ea779a-ccc0-44be-9f23-283f1c210221\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"
Jul 27 18:22:52.360777 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.360734    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8g5s\" (UniqueName: \"kubernetes.io/projected/b5ea779a-ccc0-44be-9f23-283f1c210221-kube-api-access-s8g5s\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6\" (UID: \"b5ea779a-ccc0-44be-9f23-283f1c210221\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"
Jul 27 18:22:52.361005 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.360821    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ea779a-ccc0-44be-9f23-283f1c210221-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6\" (UID: \"b5ea779a-ccc0-44be-9f23-283f1c210221\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"
Jul 27 18:22:52.363616 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.363582    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ea779a-ccc0-44be-9f23-283f1c210221-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6\" (UID: \"b5ea779a-ccc0-44be-9f23-283f1c210221\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"
Jul 27 18:22:52.373703 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.373678    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8g5s\" (UniqueName: \"kubernetes.io/projected/b5ea779a-ccc0-44be-9f23-283f1c210221-kube-api-access-s8g5s\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6\" (UID: \"b5ea779a-ccc0-44be-9f23-283f1c210221\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"
Jul 27 18:22:52.471133 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.471098    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"
Jul 27 18:22:52.602093 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.602036    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"]
Jul 27 18:22:52.604625 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:22:52.604558    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ea779a_ccc0_44be_9f23_283f1c210221.slice/crio-250157d577cb911179287b181ebb076e29a4daf79e1e63a8428a4f3b0e50f5ec WatchSource:0}: Error finding container 250157d577cb911179287b181ebb076e29a4daf79e1e63a8428a4f3b0e50f5ec: Status 404 returned error can't find the container with id 250157d577cb911179287b181ebb076e29a4daf79e1e63a8428a4f3b0e50f5ec
Jul 27 18:22:52.749071 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.748972    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6" event={"ID":"b5ea779a-ccc0-44be-9f23-283f1c210221","Type":"ContainerStarted","Data":"b143bc613204929d3e2ca935cef7e00cf73a731eae92921787ad917e596829f8"}
Jul 27 18:22:52.749071 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.749052    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6" event={"ID":"b5ea779a-ccc0-44be-9f23-283f1c210221","Type":"ContainerStarted","Data":"250157d577cb911179287b181ebb076e29a4daf79e1e63a8428a4f3b0e50f5ec"}
Jul 27 18:22:52.749305 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.749119    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"
Jul 27 18:22:52.767437 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:22:52.767385    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6" podStartSLOduration=0.767368574 podStartE2EDuration="767.368574ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:22:52.766349701 +0000 UTC m=+3254.811087781" watchObservedRunningTime="2026-07-27 18:22:52.767368574 +0000 UTC m=+3254.812106655"
Jul 27 18:23:23.756162 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:23.756130    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"
Jul 27 18:23:30.109753 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.109711    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"]
Jul 27 18:23:30.113596 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.113567    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"
Jul 27 18:23:30.117188 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.117159    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 27 18:23:30.117332 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.117174    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-bjwnq\""
Jul 27 18:23:30.122303 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.122280    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"]
Jul 27 18:23:30.275226 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.275189    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fa2acd-d8d3-470d-9406-b750706fd129-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn\" (UID: \"c8fa2acd-d8d3-470d-9406-b750706fd129\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"
Jul 27 18:23:30.275410 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.275243    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bkxx\" (UniqueName: \"kubernetes.io/projected/c8fa2acd-d8d3-470d-9406-b750706fd129-kube-api-access-6bkxx\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn\" (UID: \"c8fa2acd-d8d3-470d-9406-b750706fd129\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"
Jul 27 18:23:30.376424 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.376320    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fa2acd-d8d3-470d-9406-b750706fd129-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn\" (UID: \"c8fa2acd-d8d3-470d-9406-b750706fd129\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"
Jul 27 18:23:30.376424 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.376381    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkxx\" (UniqueName: \"kubernetes.io/projected/c8fa2acd-d8d3-470d-9406-b750706fd129-kube-api-access-6bkxx\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn\" (UID: \"c8fa2acd-d8d3-470d-9406-b750706fd129\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"
Jul 27 18:23:30.378882 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.378859    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fa2acd-d8d3-470d-9406-b750706fd129-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn\" (UID: \"c8fa2acd-d8d3-470d-9406-b750706fd129\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"
Jul 27 18:23:30.384051 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.384018    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkxx\" (UniqueName: \"kubernetes.io/projected/c8fa2acd-d8d3-470d-9406-b750706fd129-kube-api-access-6bkxx\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn\" (UID: \"c8fa2acd-d8d3-470d-9406-b750706fd129\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"
Jul 27 18:23:30.426086 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.426058    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"
Jul 27 18:23:30.555978 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.555950    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"]
Jul 27 18:23:30.558372 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:23:30.558338    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fa2acd_d8d3_470d_9406_b750706fd129.slice/crio-a33be41020eef96754fa5d88abdd18dbbc0cc3c7a0cab063986f724ffb62886a WatchSource:0}: Error finding container a33be41020eef96754fa5d88abdd18dbbc0cc3c7a0cab063986f724ffb62886a: Status 404 returned error can't find the container with id a33be41020eef96754fa5d88abdd18dbbc0cc3c7a0cab063986f724ffb62886a
Jul 27 18:23:30.934893 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.934853    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn" event={"ID":"c8fa2acd-d8d3-470d-9406-b750706fd129","Type":"ContainerStarted","Data":"92d522d589b22e2840b4d703fabdf839fa2a9e4729a28709da8d8a68009ba924"}
Jul 27 18:23:30.935104 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.934942    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn" event={"ID":"c8fa2acd-d8d3-470d-9406-b750706fd129","Type":"ContainerStarted","Data":"a33be41020eef96754fa5d88abdd18dbbc0cc3c7a0cab063986f724ffb62886a"}
Jul 27 18:23:30.935104 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.934972    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"
Jul 27 18:23:30.956055 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:23:30.956006    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn" podStartSLOduration=0.955989831 podStartE2EDuration="955.989831ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:23:30.954896652 +0000 UTC m=+3292.999634733" watchObservedRunningTime="2026-07-27 18:23:30.955989831 +0000 UTC m=+3293.000727912"
Jul 27 18:24:01.941913 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:01.941833    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"
Jul 27 18:24:03.629910 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:03.629874    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"]
Jul 27 18:24:11.941643 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:11.941598    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn" podUID="c8fa2acd-d8d3-470d-9406-b750706fd129" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:24:18.649199 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:18.649128    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn" podUID="c8fa2acd-d8d3-470d-9406-b750706fd129" containerName="main" containerID="cri-o://92d522d589b22e2840b4d703fabdf839fa2a9e4729a28709da8d8a68009ba924" gracePeriod=45
Jul 27 18:24:18.891659 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:18.891634    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"
Jul 27 18:24:18.991151 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:18.991118    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bkxx\" (UniqueName: \"kubernetes.io/projected/c8fa2acd-d8d3-470d-9406-b750706fd129-kube-api-access-6bkxx\") pod \"c8fa2acd-d8d3-470d-9406-b750706fd129\" (UID: \"c8fa2acd-d8d3-470d-9406-b750706fd129\") "
Jul 27 18:24:18.991361 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:18.991176    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fa2acd-d8d3-470d-9406-b750706fd129-tls-certs\") pod \"c8fa2acd-d8d3-470d-9406-b750706fd129\" (UID: \"c8fa2acd-d8d3-470d-9406-b750706fd129\") "
Jul 27 18:24:18.993369 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:18.993332    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8fa2acd-d8d3-470d-9406-b750706fd129-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8fa2acd-d8d3-470d-9406-b750706fd129" (UID: "c8fa2acd-d8d3-470d-9406-b750706fd129"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:24:18.993479 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:18.993385    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fa2acd-d8d3-470d-9406-b750706fd129-kube-api-access-6bkxx" (OuterVolumeSpecName: "kube-api-access-6bkxx") pod "c8fa2acd-d8d3-470d-9406-b750706fd129" (UID: "c8fa2acd-d8d3-470d-9406-b750706fd129"). InnerVolumeSpecName "kube-api-access-6bkxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:24:19.092172 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:19.092133    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fa2acd-d8d3-470d-9406-b750706fd129-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:24:19.092172 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:19.092166    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bkxx\" (UniqueName: \"kubernetes.io/projected/c8fa2acd-d8d3-470d-9406-b750706fd129-kube-api-access-6bkxx\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:24:19.170153 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:19.170123    2571 generic.go:356] "Generic (PLEG): container finished" podID="c8fa2acd-d8d3-470d-9406-b750706fd129" containerID="92d522d589b22e2840b4d703fabdf839fa2a9e4729a28709da8d8a68009ba924" exitCode=0
Jul 27 18:24:19.170305 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:19.170165    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn" event={"ID":"c8fa2acd-d8d3-470d-9406-b750706fd129","Type":"ContainerDied","Data":"92d522d589b22e2840b4d703fabdf839fa2a9e4729a28709da8d8a68009ba924"}
Jul 27 18:24:19.170305 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:19.170171    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"
Jul 27 18:24:19.170305 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:19.170207    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn" event={"ID":"c8fa2acd-d8d3-470d-9406-b750706fd129","Type":"ContainerDied","Data":"a33be41020eef96754fa5d88abdd18dbbc0cc3c7a0cab063986f724ffb62886a"}
Jul 27 18:24:19.170305 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:19.170232    2571 scope.go:117] "RemoveContainer" containerID="92d522d589b22e2840b4d703fabdf839fa2a9e4729a28709da8d8a68009ba924"
Jul 27 18:24:19.179579 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:19.179556    2571 scope.go:117] "RemoveContainer" containerID="92d522d589b22e2840b4d703fabdf839fa2a9e4729a28709da8d8a68009ba924"
Jul 27 18:24:19.179859 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:24:19.179837    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d522d589b22e2840b4d703fabdf839fa2a9e4729a28709da8d8a68009ba924\": container with ID starting with 92d522d589b22e2840b4d703fabdf839fa2a9e4729a28709da8d8a68009ba924 not found: ID does not exist" containerID="92d522d589b22e2840b4d703fabdf839fa2a9e4729a28709da8d8a68009ba924"
Jul 27 18:24:19.179929 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:19.179867    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d522d589b22e2840b4d703fabdf839fa2a9e4729a28709da8d8a68009ba924"} err="failed to get container status \"92d522d589b22e2840b4d703fabdf839fa2a9e4729a28709da8d8a68009ba924\": rpc error: code = NotFound desc = could not find container \"92d522d589b22e2840b4d703fabdf839fa2a9e4729a28709da8d8a68009ba924\": container with ID starting with 92d522d589b22e2840b4d703fabdf839fa2a9e4729a28709da8d8a68009ba924 not found: ID does not exist"
Jul 27 18:24:19.195114 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:19.195090    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"]
Jul 27 18:24:19.198771 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:19.198745    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche6kxnn"]
Jul 27 18:24:20.399285 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:20.399252    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8fa2acd-d8d3-470d-9406-b750706fd129" path="/var/lib/kubelet/pods/c8fa2acd-d8d3-470d-9406-b750706fd129/volumes"
Jul 27 18:24:24.002013 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:24.001975    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"]
Jul 27 18:24:33.755810 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:33.755770    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6" podUID="b5ea779a-ccc0-44be-9f23-283f1c210221" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:24:39.026975 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.026845    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6" podUID="b5ea779a-ccc0-44be-9f23-283f1c210221" containerName="main" containerID="cri-o://b143bc613204929d3e2ca935cef7e00cf73a731eae92921787ad917e596829f8" gracePeriod=45
Jul 27 18:24:39.273975 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.273951    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"
Jul 27 18:24:39.276031 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.276004    2571 generic.go:356] "Generic (PLEG): container finished" podID="b5ea779a-ccc0-44be-9f23-283f1c210221" containerID="b143bc613204929d3e2ca935cef7e00cf73a731eae92921787ad917e596829f8" exitCode=0
Jul 27 18:24:39.276177 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.276040    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"
Jul 27 18:24:39.276177 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.276048    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6" event={"ID":"b5ea779a-ccc0-44be-9f23-283f1c210221","Type":"ContainerDied","Data":"b143bc613204929d3e2ca935cef7e00cf73a731eae92921787ad917e596829f8"}
Jul 27 18:24:39.276177 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.276150    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6" event={"ID":"b5ea779a-ccc0-44be-9f23-283f1c210221","Type":"ContainerDied","Data":"250157d577cb911179287b181ebb076e29a4daf79e1e63a8428a4f3b0e50f5ec"}
Jul 27 18:24:39.276177 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.276171    2571 scope.go:117] "RemoveContainer" containerID="b143bc613204929d3e2ca935cef7e00cf73a731eae92921787ad917e596829f8"
Jul 27 18:24:39.284562 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.284545    2571 scope.go:117] "RemoveContainer" containerID="b143bc613204929d3e2ca935cef7e00cf73a731eae92921787ad917e596829f8"
Jul 27 18:24:39.284822 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:24:39.284795    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b143bc613204929d3e2ca935cef7e00cf73a731eae92921787ad917e596829f8\": container with ID starting with b143bc613204929d3e2ca935cef7e00cf73a731eae92921787ad917e596829f8 not found: ID does not exist" containerID="b143bc613204929d3e2ca935cef7e00cf73a731eae92921787ad917e596829f8"
Jul 27 18:24:39.284923 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.284829    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b143bc613204929d3e2ca935cef7e00cf73a731eae92921787ad917e596829f8"} err="failed to get container status \"b143bc613204929d3e2ca935cef7e00cf73a731eae92921787ad917e596829f8\": rpc error: code = NotFound desc = could not find container \"b143bc613204929d3e2ca935cef7e00cf73a731eae92921787ad917e596829f8\": container with ID starting with b143bc613204929d3e2ca935cef7e00cf73a731eae92921787ad917e596829f8 not found: ID does not exist"
Jul 27 18:24:39.371515 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.371452    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8g5s\" (UniqueName: \"kubernetes.io/projected/b5ea779a-ccc0-44be-9f23-283f1c210221-kube-api-access-s8g5s\") pod \"b5ea779a-ccc0-44be-9f23-283f1c210221\" (UID: \"b5ea779a-ccc0-44be-9f23-283f1c210221\") "
Jul 27 18:24:39.371715 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.371529    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ea779a-ccc0-44be-9f23-283f1c210221-tls-certs\") pod \"b5ea779a-ccc0-44be-9f23-283f1c210221\" (UID: \"b5ea779a-ccc0-44be-9f23-283f1c210221\") "
Jul 27 18:24:39.373667 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.373637    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ea779a-ccc0-44be-9f23-283f1c210221-kube-api-access-s8g5s" (OuterVolumeSpecName: "kube-api-access-s8g5s") pod "b5ea779a-ccc0-44be-9f23-283f1c210221" (UID: "b5ea779a-ccc0-44be-9f23-283f1c210221"). InnerVolumeSpecName "kube-api-access-s8g5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:24:39.373667 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.373658    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5ea779a-ccc0-44be-9f23-283f1c210221-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5ea779a-ccc0-44be-9f23-283f1c210221" (UID: "b5ea779a-ccc0-44be-9f23-283f1c210221"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:24:39.473024 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.472992    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8g5s\" (UniqueName: \"kubernetes.io/projected/b5ea779a-ccc0-44be-9f23-283f1c210221-kube-api-access-s8g5s\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:24:39.473024 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.473019    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ea779a-ccc0-44be-9f23-283f1c210221-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:24:39.602524 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.602479    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"]
Jul 27 18:24:39.606739 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:39.606712    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetndd6"]
Jul 27 18:24:40.404157 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:40.404120    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5ea779a-ccc0-44be-9f23-283f1c210221" path="/var/lib/kubelet/pods/b5ea779a-ccc0-44be-9f23-283f1c210221/volumes"
Jul 27 18:24:48.168513 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.168456    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"]
Jul 27 18:24:48.169002 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.168941    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5ea779a-ccc0-44be-9f23-283f1c210221" containerName="main"
Jul 27 18:24:48.169002 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.168959    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5ea779a-ccc0-44be-9f23-283f1c210221" containerName="main"
Jul 27 18:24:48.169002 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.168970    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8fa2acd-d8d3-470d-9406-b750706fd129" containerName="main"
Jul 27 18:24:48.169002 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.168978    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8fa2acd-d8d3-470d-9406-b750706fd129" containerName="main"
Jul 27 18:24:48.169220 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.169057    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8fa2acd-d8d3-470d-9406-b750706fd129" containerName="main"
Jul 27 18:24:48.169220 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.169072    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5ea779a-ccc0-44be-9f23-283f1c210221" containerName="main"
Jul 27 18:24:48.173556 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.173535    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"
Jul 27 18:24:48.176133 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.176111    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 27 18:24:48.177391 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.177309    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 27 18:24:48.177391 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.177312    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 27 18:24:48.177640 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.177606    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-cpnsd\""
Jul 27 18:24:48.177737 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.177720    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-wbgw5\""
Jul 27 18:24:48.180795 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.180770    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"]
Jul 27 18:24:48.240385 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.240350    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07dc3c58-4539-4019-b3a0-64db433cebf9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7\" (UID: \"07dc3c58-4539-4019-b3a0-64db433cebf9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"
Jul 27 18:24:48.240583 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.240402    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbndb\" (UniqueName: \"kubernetes.io/projected/07dc3c58-4539-4019-b3a0-64db433cebf9-kube-api-access-fbndb\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7\" (UID: \"07dc3c58-4539-4019-b3a0-64db433cebf9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"
Jul 27 18:24:48.341121 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.341085    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07dc3c58-4539-4019-b3a0-64db433cebf9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7\" (UID: \"07dc3c58-4539-4019-b3a0-64db433cebf9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"
Jul 27 18:24:48.341289 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.341130    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbndb\" (UniqueName: \"kubernetes.io/projected/07dc3c58-4539-4019-b3a0-64db433cebf9-kube-api-access-fbndb\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7\" (UID: \"07dc3c58-4539-4019-b3a0-64db433cebf9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"
Jul 27 18:24:48.343653 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.343629    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07dc3c58-4539-4019-b3a0-64db433cebf9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7\" (UID: \"07dc3c58-4539-4019-b3a0-64db433cebf9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"
Jul 27 18:24:48.353751 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.353721    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbndb\" (UniqueName: \"kubernetes.io/projected/07dc3c58-4539-4019-b3a0-64db433cebf9-kube-api-access-fbndb\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7\" (UID: \"07dc3c58-4539-4019-b3a0-64db433cebf9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"
Jul 27 18:24:48.484314 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.484278    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"
Jul 27 18:24:48.613263 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:48.613237    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"]
Jul 27 18:24:48.615464 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:24:48.615434    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07dc3c58_4539_4019_b3a0_64db433cebf9.slice/crio-03a07a73563a335c444aa7e8c6d8df6be7d916a0c21d6f0158e8144644151c3c WatchSource:0}: Error finding container 03a07a73563a335c444aa7e8c6d8df6be7d916a0c21d6f0158e8144644151c3c: Status 404 returned error can't find the container with id 03a07a73563a335c444aa7e8c6d8df6be7d916a0c21d6f0158e8144644151c3c
Jul 27 18:24:49.325225 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:49.325175    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7" event={"ID":"07dc3c58-4539-4019-b3a0-64db433cebf9","Type":"ContainerStarted","Data":"3e7bb55dce4f81f79e395d98c74485f315af3b8822efc295ad63d45ae19f5c7e"}
Jul 27 18:24:49.325708 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:49.325264    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"
Jul 27 18:24:49.325708 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:49.325281    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7" event={"ID":"07dc3c58-4539-4019-b3a0-64db433cebf9","Type":"ContainerStarted","Data":"03a07a73563a335c444aa7e8c6d8df6be7d916a0c21d6f0158e8144644151c3c"}
Jul 27 18:24:49.345623 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:24:49.345566    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7" podStartSLOduration=1.345546881 podStartE2EDuration="1.345546881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:24:49.3421556 +0000 UTC m=+3371.386893682" watchObservedRunningTime="2026-07-27 18:24:49.345546881 +0000 UTC m=+3371.390284963"
Jul 27 18:25:20.332076 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:25:20.332044    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"
Jul 27 18:26:43.297275 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:43.297229    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"]
Jul 27 18:26:50.331523 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:50.331463    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7" podUID="07dc3c58-4539-4019-b3a0-64db433cebf9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:26:50.868711 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:50.868667    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"]
Jul 27 18:26:50.877256 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:26:50.877216    2571 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 18:26:50.877256 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:26:50.877256 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:26:50.877256 ip-10-0-131-74 kubenswrapper[2571]:  > containerID="84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2" cmd=["/bin/sleep","15"]
Jul 27 18:26:50.877572 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:50.877283    2571 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l" message="" err=<
Jul 27 18:26:50.877572 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:26:50.877572 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:26:50.877572 ip-10-0-131-74 kubenswrapper[2571]:  >
Jul 27 18:26:50.877572 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:26:50.877299    2571 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 18:26:50.877572 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:26:50.877572 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:26:50.877572 ip-10-0-131-74 kubenswrapper[2571]:  > pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l" podUID="65452329-e447-4dc1-8119-f124c30e0246" containerName="main" containerID="cri-o://84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2"
Jul 27 18:26:50.877572 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:50.877346    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l" podUID="65452329-e447-4dc1-8119-f124c30e0246" containerName="main" containerID="cri-o://84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2" gracePeriod=60
Jul 27 18:26:51.229986 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.229960    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:26:51.295509 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.295460    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j67h\" (UniqueName: \"kubernetes.io/projected/65452329-e447-4dc1-8119-f124c30e0246-kube-api-access-6j67h\") pod \"65452329-e447-4dc1-8119-f124c30e0246\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") "
Jul 27 18:26:51.295664 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.295544    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65452329-e447-4dc1-8119-f124c30e0246-tls-certs\") pod \"65452329-e447-4dc1-8119-f124c30e0246\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") "
Jul 27 18:26:51.295664 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.295586    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-model-cache\") pod \"65452329-e447-4dc1-8119-f124c30e0246\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") "
Jul 27 18:26:51.295664 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.295638    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-tmp-dir\") pod \"65452329-e447-4dc1-8119-f124c30e0246\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") "
Jul 27 18:26:51.295890 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.295672    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-home\") pod \"65452329-e447-4dc1-8119-f124c30e0246\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") "
Jul 27 18:26:51.295890 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.295697    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-dshm\") pod \"65452329-e447-4dc1-8119-f124c30e0246\" (UID: \"65452329-e447-4dc1-8119-f124c30e0246\") "
Jul 27 18:26:51.296009 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.295907    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-model-cache" (OuterVolumeSpecName: "model-cache") pod "65452329-e447-4dc1-8119-f124c30e0246" (UID: "65452329-e447-4dc1-8119-f124c30e0246"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:26:51.296009 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.295920    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-home" (OuterVolumeSpecName: "home") pod "65452329-e447-4dc1-8119-f124c30e0246" (UID: "65452329-e447-4dc1-8119-f124c30e0246"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:26:51.296009 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.295947    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65452329-e447-4dc1-8119-f124c30e0246" (UID: "65452329-e447-4dc1-8119-f124c30e0246"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:26:51.297820 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.297797    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65452329-e447-4dc1-8119-f124c30e0246-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65452329-e447-4dc1-8119-f124c30e0246" (UID: "65452329-e447-4dc1-8119-f124c30e0246"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:26:51.297936 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.297862    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65452329-e447-4dc1-8119-f124c30e0246-kube-api-access-6j67h" (OuterVolumeSpecName: "kube-api-access-6j67h") pod "65452329-e447-4dc1-8119-f124c30e0246" (UID: "65452329-e447-4dc1-8119-f124c30e0246"). InnerVolumeSpecName "kube-api-access-6j67h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:26:51.297936 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.297876    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-dshm" (OuterVolumeSpecName: "dshm") pod "65452329-e447-4dc1-8119-f124c30e0246" (UID: "65452329-e447-4dc1-8119-f124c30e0246"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:26:51.396357 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.396268    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j67h\" (UniqueName: \"kubernetes.io/projected/65452329-e447-4dc1-8119-f124c30e0246-kube-api-access-6j67h\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:26:51.396357 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.396298    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65452329-e447-4dc1-8119-f124c30e0246-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:26:51.396357 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.396314    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:26:51.396357 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.396325    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:26:51.396357 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.396334    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:26:51.396357 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.396345    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65452329-e447-4dc1-8119-f124c30e0246-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:26:51.973607 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.973570    2571 generic.go:356] "Generic (PLEG): container finished" podID="65452329-e447-4dc1-8119-f124c30e0246" containerID="84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2" exitCode=0
Jul 27 18:26:51.973785 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.973622    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"
Jul 27 18:26:51.973785 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.973658    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l" event={"ID":"65452329-e447-4dc1-8119-f124c30e0246","Type":"ContainerDied","Data":"84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2"}
Jul 27 18:26:51.973785 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.973718    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l" event={"ID":"65452329-e447-4dc1-8119-f124c30e0246","Type":"ContainerDied","Data":"772c5751ec4278ce4ae1996fb7d558d82c3c2fe428216a31cd1486e4275dce01"}
Jul 27 18:26:51.973785 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.973744    2571 scope.go:117] "RemoveContainer" containerID="84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2"
Jul 27 18:26:51.983029 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.983009    2571 scope.go:117] "RemoveContainer" containerID="84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2"
Jul 27 18:26:51.983308 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:26:51.983284    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2\": container with ID starting with 84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2 not found: ID does not exist" containerID="84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2"
Jul 27 18:26:51.983399 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.983312    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2"} err="failed to get container status \"84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2\": rpc error: code = NotFound desc = could not find container \"84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2\": container with ID starting with 84950980f6a96bf4ab8cf3c2280b5bba0aed4df64ec7fff369e6ed13c84cf6a2 not found: ID does not exist"
Jul 27 18:26:51.995896 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.995874    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"]
Jul 27 18:26:51.998592 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:51.998569    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l"]
Jul 27 18:26:52.399855 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:52.399763    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65452329-e447-4dc1-8119-f124c30e0246" path="/var/lib/kubelet/pods/65452329-e447-4dc1-8119-f124c30e0246/volumes"
Jul 27 18:26:58.316368 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:58.316329    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7" podUID="07dc3c58-4539-4019-b3a0-64db433cebf9" containerName="main" containerID="cri-o://3e7bb55dce4f81f79e395d98c74485f315af3b8822efc295ad63d45ae19f5c7e" gracePeriod=45
Jul 27 18:26:58.566117 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:58.566091    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"
Jul 27 18:26:58.651545 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:58.651439    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07dc3c58-4539-4019-b3a0-64db433cebf9-tls-certs\") pod \"07dc3c58-4539-4019-b3a0-64db433cebf9\" (UID: \"07dc3c58-4539-4019-b3a0-64db433cebf9\") "
Jul 27 18:26:58.651545 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:58.651535    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbndb\" (UniqueName: \"kubernetes.io/projected/07dc3c58-4539-4019-b3a0-64db433cebf9-kube-api-access-fbndb\") pod \"07dc3c58-4539-4019-b3a0-64db433cebf9\" (UID: \"07dc3c58-4539-4019-b3a0-64db433cebf9\") "
Jul 27 18:26:58.653673 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:58.653643    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07dc3c58-4539-4019-b3a0-64db433cebf9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07dc3c58-4539-4019-b3a0-64db433cebf9" (UID: "07dc3c58-4539-4019-b3a0-64db433cebf9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:26:58.653816 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:58.653730    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07dc3c58-4539-4019-b3a0-64db433cebf9-kube-api-access-fbndb" (OuterVolumeSpecName: "kube-api-access-fbndb") pod "07dc3c58-4539-4019-b3a0-64db433cebf9" (UID: "07dc3c58-4539-4019-b3a0-64db433cebf9"). InnerVolumeSpecName "kube-api-access-fbndb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:26:58.752366 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:58.752334    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07dc3c58-4539-4019-b3a0-64db433cebf9-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:26:58.752366 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:58.752363    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbndb\" (UniqueName: \"kubernetes.io/projected/07dc3c58-4539-4019-b3a0-64db433cebf9-kube-api-access-fbndb\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:26:59.010066 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:59.010031    2571 generic.go:356] "Generic (PLEG): container finished" podID="07dc3c58-4539-4019-b3a0-64db433cebf9" containerID="3e7bb55dce4f81f79e395d98c74485f315af3b8822efc295ad63d45ae19f5c7e" exitCode=0
Jul 27 18:26:59.010285 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:59.010074    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"
Jul 27 18:26:59.010285 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:59.010124    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7" event={"ID":"07dc3c58-4539-4019-b3a0-64db433cebf9","Type":"ContainerDied","Data":"3e7bb55dce4f81f79e395d98c74485f315af3b8822efc295ad63d45ae19f5c7e"}
Jul 27 18:26:59.010285 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:59.010184    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7" event={"ID":"07dc3c58-4539-4019-b3a0-64db433cebf9","Type":"ContainerDied","Data":"03a07a73563a335c444aa7e8c6d8df6be7d916a0c21d6f0158e8144644151c3c"}
Jul 27 18:26:59.010285 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:59.010214    2571 scope.go:117] "RemoveContainer" containerID="3e7bb55dce4f81f79e395d98c74485f315af3b8822efc295ad63d45ae19f5c7e"
Jul 27 18:26:59.019179 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:59.019164    2571 scope.go:117] "RemoveContainer" containerID="3e7bb55dce4f81f79e395d98c74485f315af3b8822efc295ad63d45ae19f5c7e"
Jul 27 18:26:59.019443 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:26:59.019423    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e7bb55dce4f81f79e395d98c74485f315af3b8822efc295ad63d45ae19f5c7e\": container with ID starting with 3e7bb55dce4f81f79e395d98c74485f315af3b8822efc295ad63d45ae19f5c7e not found: ID does not exist" containerID="3e7bb55dce4f81f79e395d98c74485f315af3b8822efc295ad63d45ae19f5c7e"
Jul 27 18:26:59.019507 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:59.019451    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7bb55dce4f81f79e395d98c74485f315af3b8822efc295ad63d45ae19f5c7e"} err="failed to get container status \"3e7bb55dce4f81f79e395d98c74485f315af3b8822efc295ad63d45ae19f5c7e\": rpc error: code = NotFound desc = could not find container \"3e7bb55dce4f81f79e395d98c74485f315af3b8822efc295ad63d45ae19f5c7e\": container with ID starting with 3e7bb55dce4f81f79e395d98c74485f315af3b8822efc295ad63d45ae19f5c7e not found: ID does not exist"
Jul 27 18:26:59.031708 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:59.031677    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"]
Jul 27 18:26:59.036613 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:26:59.036590    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh5zk7"]
Jul 27 18:27:00.398321 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:00.398290    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07dc3c58-4539-4019-b3a0-64db433cebf9" path="/var/lib/kubelet/pods/07dc3c58-4539-4019-b3a0-64db433cebf9/volumes"
Jul 27 18:27:13.649877 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.649833    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"]
Jul 27 18:27:13.650438 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.650281    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07dc3c58-4539-4019-b3a0-64db433cebf9" containerName="main"
Jul 27 18:27:13.650438 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.650301    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="07dc3c58-4539-4019-b3a0-64db433cebf9" containerName="main"
Jul 27 18:27:13.650438 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.650319    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65452329-e447-4dc1-8119-f124c30e0246" containerName="main"
Jul 27 18:27:13.650438 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.650327    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="65452329-e447-4dc1-8119-f124c30e0246" containerName="main"
Jul 27 18:27:13.650438 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.650405    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="07dc3c58-4539-4019-b3a0-64db433cebf9" containerName="main"
Jul 27 18:27:13.650438 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.650421    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="65452329-e447-4dc1-8119-f124c30e0246" containerName="main"
Jul 27 18:27:13.654804 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.654783    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"
Jul 27 18:27:13.658669 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.658638    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-ht249\""
Jul 27 18:27:13.658669 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.658649    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 27 18:27:13.658852 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.658731    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 27 18:27:13.658852 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.658733    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 27 18:27:13.658852 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.658772    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-s78vq\""
Jul 27 18:27:13.665004 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.664980    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"]
Jul 27 18:27:13.774105 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.774064    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxwp4\" (UniqueName: \"kubernetes.io/projected/07853d27-a401-4837-9bfe-b181dbd00c27-kube-api-access-hxwp4\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5\" (UID: \"07853d27-a401-4837-9bfe-b181dbd00c27\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"
Jul 27 18:27:13.774303 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.774190    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07853d27-a401-4837-9bfe-b181dbd00c27-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5\" (UID: \"07853d27-a401-4837-9bfe-b181dbd00c27\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"
Jul 27 18:27:13.874761 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.874721    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxwp4\" (UniqueName: \"kubernetes.io/projected/07853d27-a401-4837-9bfe-b181dbd00c27-kube-api-access-hxwp4\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5\" (UID: \"07853d27-a401-4837-9bfe-b181dbd00c27\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"
Jul 27 18:27:13.874943 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.874803    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07853d27-a401-4837-9bfe-b181dbd00c27-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5\" (UID: \"07853d27-a401-4837-9bfe-b181dbd00c27\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"
Jul 27 18:27:13.877478 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.877445    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07853d27-a401-4837-9bfe-b181dbd00c27-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5\" (UID: \"07853d27-a401-4837-9bfe-b181dbd00c27\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"
Jul 27 18:27:13.885972 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.885945    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxwp4\" (UniqueName: \"kubernetes.io/projected/07853d27-a401-4837-9bfe-b181dbd00c27-kube-api-access-hxwp4\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5\" (UID: \"07853d27-a401-4837-9bfe-b181dbd00c27\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"
Jul 27 18:27:13.965654 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:13.965617    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"
Jul 27 18:27:14.099824 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:14.099789    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"]
Jul 27 18:27:14.102866 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:27:14.102839    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07853d27_a401_4837_9bfe_b181dbd00c27.slice/crio-3682ce405d9f20fcd9e1c446ae1806dbdbc5b41d05db8ddf348c8ca6a981e529 WatchSource:0}: Error finding container 3682ce405d9f20fcd9e1c446ae1806dbdbc5b41d05db8ddf348c8ca6a981e529: Status 404 returned error can't find the container with id 3682ce405d9f20fcd9e1c446ae1806dbdbc5b41d05db8ddf348c8ca6a981e529
Jul 27 18:27:15.095266 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:15.095230    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5" event={"ID":"07853d27-a401-4837-9bfe-b181dbd00c27","Type":"ContainerStarted","Data":"b3c25e9daa37605f99a29eee804c02679bd9d8b3962feb5f0b781837bae002a9"}
Jul 27 18:27:15.095790 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:15.095284    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5" event={"ID":"07853d27-a401-4837-9bfe-b181dbd00c27","Type":"ContainerStarted","Data":"3682ce405d9f20fcd9e1c446ae1806dbdbc5b41d05db8ddf348c8ca6a981e529"}
Jul 27 18:27:15.095790 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:15.095336    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"
Jul 27 18:27:15.113372 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:15.113314    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5" podStartSLOduration=2.11329571 podStartE2EDuration="2.11329571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:27:15.112368785 +0000 UTC m=+3517.157106866" watchObservedRunningTime="2026-07-27 18:27:15.11329571 +0000 UTC m=+3517.158033791"
Jul 27 18:27:21.358323 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.358289    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 27 18:27:21.361761 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.361744    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.365546 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.365521    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 27 18:27:21.365752 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.365731    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-mn-dockercfg-8th6f\""
Jul 27 18:27:21.365818 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.365786    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 27 18:27:21.365877 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.365834    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-wg99b\""
Jul 27 18:27:21.365921 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.365893    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 27 18:27:21.374088 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.374065    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 27 18:27:21.538394 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.538355    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkcnf\" (UniqueName: \"kubernetes.io/projected/cf7d1299-ccfb-439c-b411-06540586a0b8-kube-api-access-wkcnf\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.538394 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.538395    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.538699 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.538428    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.538699 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.538568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.538699 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.538605    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7d1299-ccfb-439c-b411-06540586a0b8-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.538699 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.538631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.639807 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.639701    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkcnf\" (UniqueName: \"kubernetes.io/projected/cf7d1299-ccfb-439c-b411-06540586a0b8-kube-api-access-wkcnf\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.639807 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.639753    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.639807 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.639793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.640124 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.639844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.640124 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.639866    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7d1299-ccfb-439c-b411-06540586a0b8-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.640124 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.639982    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.640283 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.640237    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.640325 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.640290    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.640360 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.640333    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.642157 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.642140    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.642360 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.642344    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7d1299-ccfb-439c-b411-06540586a0b8-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.648422 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.648399    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkcnf\" (UniqueName: \"kubernetes.io/projected/cf7d1299-ccfb-439c-b411-06540586a0b8-kube-api-access-wkcnf\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.672457 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.672433    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:27:21.810423 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:21.810293    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 27 18:27:21.813097 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:27:21.813060    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7d1299_ccfb_439c_b411_06540586a0b8.slice/crio-7d3519892582ff55208f68f414b93bb0f22974902a1a15dd66ab3c426487d30f WatchSource:0}: Error finding container 7d3519892582ff55208f68f414b93bb0f22974902a1a15dd66ab3c426487d30f: Status 404 returned error can't find the container with id 7d3519892582ff55208f68f414b93bb0f22974902a1a15dd66ab3c426487d30f
Jul 27 18:27:22.131739 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:22.131701    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"cf7d1299-ccfb-439c-b411-06540586a0b8","Type":"ContainerStarted","Data":"0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1"}
Jul 27 18:27:22.131942 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:22.131788    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"cf7d1299-ccfb-439c-b411-06540586a0b8","Type":"ContainerStarted","Data":"7d3519892582ff55208f68f414b93bb0f22974902a1a15dd66ab3c426487d30f"}
Jul 27 18:27:22.151229 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:22.151176    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podStartSLOduration=1.151162794 podStartE2EDuration="1.151162794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:27:22.149727676 +0000 UTC m=+3524.194465770" watchObservedRunningTime="2026-07-27 18:27:22.151162794 +0000 UTC m=+3524.195900875"
Jul 27 18:27:46.103203 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:27:46.103166    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"
Jul 27 18:28:45.480338 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:28:45.480303    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"]
Jul 27 18:28:46.102757 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:28:46.102712    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5" podUID="07853d27-a401-4837-9bfe-b181dbd00c27" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:28:56.102409 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:28:56.102360    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5" podUID="07853d27-a401-4837-9bfe-b181dbd00c27" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:29:00.499753 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:00.499691    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5" podUID="07853d27-a401-4837-9bfe-b181dbd00c27" containerName="main" containerID="cri-o://b3c25e9daa37605f99a29eee804c02679bd9d8b3962feb5f0b781837bae002a9" gracePeriod=45
Jul 27 18:29:00.640462 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:00.640427    2571 generic.go:356] "Generic (PLEG): container finished" podID="07853d27-a401-4837-9bfe-b181dbd00c27" containerID="b3c25e9daa37605f99a29eee804c02679bd9d8b3962feb5f0b781837bae002a9" exitCode=0
Jul 27 18:29:00.640651 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:00.640511    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5" event={"ID":"07853d27-a401-4837-9bfe-b181dbd00c27","Type":"ContainerDied","Data":"b3c25e9daa37605f99a29eee804c02679bd9d8b3962feb5f0b781837bae002a9"}
Jul 27 18:29:00.753980 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:00.753923    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"
Jul 27 18:29:00.899048 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:00.899010    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07853d27-a401-4837-9bfe-b181dbd00c27-tls-certs\") pod \"07853d27-a401-4837-9bfe-b181dbd00c27\" (UID: \"07853d27-a401-4837-9bfe-b181dbd00c27\") "
Jul 27 18:29:00.899220 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:00.899065    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxwp4\" (UniqueName: \"kubernetes.io/projected/07853d27-a401-4837-9bfe-b181dbd00c27-kube-api-access-hxwp4\") pod \"07853d27-a401-4837-9bfe-b181dbd00c27\" (UID: \"07853d27-a401-4837-9bfe-b181dbd00c27\") "
Jul 27 18:29:00.901187 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:00.901164    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07853d27-a401-4837-9bfe-b181dbd00c27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07853d27-a401-4837-9bfe-b181dbd00c27" (UID: "07853d27-a401-4837-9bfe-b181dbd00c27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:29:00.901304 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:00.901289    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07853d27-a401-4837-9bfe-b181dbd00c27-kube-api-access-hxwp4" (OuterVolumeSpecName: "kube-api-access-hxwp4") pod "07853d27-a401-4837-9bfe-b181dbd00c27" (UID: "07853d27-a401-4837-9bfe-b181dbd00c27"). InnerVolumeSpecName "kube-api-access-hxwp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:29:00.999753 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:00.999715    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07853d27-a401-4837-9bfe-b181dbd00c27-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:29:00.999753 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:00.999747    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxwp4\" (UniqueName: \"kubernetes.io/projected/07853d27-a401-4837-9bfe-b181dbd00c27-kube-api-access-hxwp4\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.646543 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:01.646479    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5" event={"ID":"07853d27-a401-4837-9bfe-b181dbd00c27","Type":"ContainerDied","Data":"3682ce405d9f20fcd9e1c446ae1806dbdbc5b41d05db8ddf348c8ca6a981e529"}
Jul 27 18:29:01.646543 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:01.646528    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"
Jul 27 18:29:01.647071 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:01.646582    2571 scope.go:117] "RemoveContainer" containerID="b3c25e9daa37605f99a29eee804c02679bd9d8b3962feb5f0b781837bae002a9"
Jul 27 18:29:01.668983 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:01.668954    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"]
Jul 27 18:29:01.673936 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:01.673914    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schegpkx5"]
Jul 27 18:29:02.398603 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:02.398567    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07853d27-a401-4837-9bfe-b181dbd00c27" path="/var/lib/kubelet/pods/07853d27-a401-4837-9bfe-b181dbd00c27/volumes"
Jul 27 18:29:31.690230 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.690197    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"]
Jul 27 18:29:31.690744 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.690545    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07853d27-a401-4837-9bfe-b181dbd00c27" containerName="main"
Jul 27 18:29:31.690744 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.690558    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="07853d27-a401-4837-9bfe-b181dbd00c27" containerName="main"
Jul 27 18:29:31.690744 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.690615    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="07853d27-a401-4837-9bfe-b181dbd00c27" containerName="main"
Jul 27 18:29:31.693548 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.693528    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"
Jul 27 18:29:31.697575 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.697546    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 27 18:29:31.697575 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.697557    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-gz7n6\""
Jul 27 18:29:31.697575 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.697584    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 27 18:29:31.697831 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.697549    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 27 18:29:31.697831 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.697698    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-c7snk\""
Jul 27 18:29:31.701741 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.701258    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"]
Jul 27 18:29:31.721664 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.721627    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47329b8-bd15-4554-bf44-c5f825292c41-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm\" (UID: \"a47329b8-bd15-4554-bf44-c5f825292c41\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"
Jul 27 18:29:31.721664 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.721662    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qshx4\" (UniqueName: \"kubernetes.io/projected/a47329b8-bd15-4554-bf44-c5f825292c41-kube-api-access-qshx4\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm\" (UID: \"a47329b8-bd15-4554-bf44-c5f825292c41\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"
Jul 27 18:29:31.822046 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.822015    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47329b8-bd15-4554-bf44-c5f825292c41-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm\" (UID: \"a47329b8-bd15-4554-bf44-c5f825292c41\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"
Jul 27 18:29:31.822046 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.822051    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qshx4\" (UniqueName: \"kubernetes.io/projected/a47329b8-bd15-4554-bf44-c5f825292c41-kube-api-access-qshx4\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm\" (UID: \"a47329b8-bd15-4554-bf44-c5f825292c41\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"
Jul 27 18:29:31.824647 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.824621    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47329b8-bd15-4554-bf44-c5f825292c41-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm\" (UID: \"a47329b8-bd15-4554-bf44-c5f825292c41\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"
Jul 27 18:29:31.830037 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:31.830009    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qshx4\" (UniqueName: \"kubernetes.io/projected/a47329b8-bd15-4554-bf44-c5f825292c41-kube-api-access-qshx4\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm\" (UID: \"a47329b8-bd15-4554-bf44-c5f825292c41\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"
Jul 27 18:29:32.007128 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:32.007094    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"
Jul 27 18:29:32.139440 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:32.139414    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"]
Jul 27 18:29:32.141748 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:29:32.141716    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47329b8_bd15_4554_bf44_c5f825292c41.slice/crio-f91e8a22af4dfbbe8c6dca1035c826f1b507bd623f6bc8856891a8cfa003a849 WatchSource:0}: Error finding container f91e8a22af4dfbbe8c6dca1035c826f1b507bd623f6bc8856891a8cfa003a849: Status 404 returned error can't find the container with id f91e8a22af4dfbbe8c6dca1035c826f1b507bd623f6bc8856891a8cfa003a849
Jul 27 18:29:32.799271 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:32.799231    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm" event={"ID":"a47329b8-bd15-4554-bf44-c5f825292c41","Type":"ContainerStarted","Data":"31c1d1d60e94fe1f407091291f29c6f8a90ba8995c5591604135c325e0444a51"}
Jul 27 18:29:32.799795 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:32.799290    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"
Jul 27 18:29:32.799795 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:32.799309    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm" event={"ID":"a47329b8-bd15-4554-bf44-c5f825292c41","Type":"ContainerStarted","Data":"f91e8a22af4dfbbe8c6dca1035c826f1b507bd623f6bc8856891a8cfa003a849"}
Jul 27 18:29:32.815167 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:29:32.815115    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm" podStartSLOduration=1.815098578 podStartE2EDuration="1.815098578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:32.814924469 +0000 UTC m=+3654.859662550" watchObservedRunningTime="2026-07-27 18:29:32.815098578 +0000 UTC m=+3654.859836660"
Jul 27 18:30:03.806366 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:30:03.806285    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"
Jul 27 18:31:46.569659 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:31:46.569566    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"]
Jul 27 18:31:53.807104 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:31:53.807059    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm" podUID="a47329b8-bd15-4554-bf44-c5f825292c41" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:32:01.591864 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:01.591819    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm" podUID="a47329b8-bd15-4554-bf44-c5f825292c41" containerName="main" containerID="cri-o://31c1d1d60e94fe1f407091291f29c6f8a90ba8995c5591604135c325e0444a51" gracePeriod=45
Jul 27 18:32:01.837995 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:01.837970    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"
Jul 27 18:32:01.965898 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:01.965868    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47329b8-bd15-4554-bf44-c5f825292c41-tls-certs\") pod \"a47329b8-bd15-4554-bf44-c5f825292c41\" (UID: \"a47329b8-bd15-4554-bf44-c5f825292c41\") "
Jul 27 18:32:01.966068 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:01.965970    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qshx4\" (UniqueName: \"kubernetes.io/projected/a47329b8-bd15-4554-bf44-c5f825292c41-kube-api-access-qshx4\") pod \"a47329b8-bd15-4554-bf44-c5f825292c41\" (UID: \"a47329b8-bd15-4554-bf44-c5f825292c41\") "
Jul 27 18:32:01.968162 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:01.968130    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a47329b8-bd15-4554-bf44-c5f825292c41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a47329b8-bd15-4554-bf44-c5f825292c41" (UID: "a47329b8-bd15-4554-bf44-c5f825292c41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:32:01.968271 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:01.968207    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47329b8-bd15-4554-bf44-c5f825292c41-kube-api-access-qshx4" (OuterVolumeSpecName: "kube-api-access-qshx4") pod "a47329b8-bd15-4554-bf44-c5f825292c41" (UID: "a47329b8-bd15-4554-bf44-c5f825292c41"). InnerVolumeSpecName "kube-api-access-qshx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:32:02.067431 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:02.067392    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qshx4\" (UniqueName: \"kubernetes.io/projected/a47329b8-bd15-4554-bf44-c5f825292c41-kube-api-access-qshx4\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:32:02.067431 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:02.067422    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47329b8-bd15-4554-bf44-c5f825292c41-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:32:02.531903 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:02.531868    2571 generic.go:356] "Generic (PLEG): container finished" podID="a47329b8-bd15-4554-bf44-c5f825292c41" containerID="31c1d1d60e94fe1f407091291f29c6f8a90ba8995c5591604135c325e0444a51" exitCode=0
Jul 27 18:32:02.532081 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:02.531910    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"
Jul 27 18:32:02.532081 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:02.531954    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm" event={"ID":"a47329b8-bd15-4554-bf44-c5f825292c41","Type":"ContainerDied","Data":"31c1d1d60e94fe1f407091291f29c6f8a90ba8995c5591604135c325e0444a51"}
Jul 27 18:32:02.532081 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:02.532008    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm" event={"ID":"a47329b8-bd15-4554-bf44-c5f825292c41","Type":"ContainerDied","Data":"f91e8a22af4dfbbe8c6dca1035c826f1b507bd623f6bc8856891a8cfa003a849"}
Jul 27 18:32:02.532081 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:02.532031    2571 scope.go:117] "RemoveContainer" containerID="31c1d1d60e94fe1f407091291f29c6f8a90ba8995c5591604135c325e0444a51"
Jul 27 18:32:02.540822 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:02.540805    2571 scope.go:117] "RemoveContainer" containerID="31c1d1d60e94fe1f407091291f29c6f8a90ba8995c5591604135c325e0444a51"
Jul 27 18:32:02.541067 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:32:02.541047    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c1d1d60e94fe1f407091291f29c6f8a90ba8995c5591604135c325e0444a51\": container with ID starting with 31c1d1d60e94fe1f407091291f29c6f8a90ba8995c5591604135c325e0444a51 not found: ID does not exist" containerID="31c1d1d60e94fe1f407091291f29c6f8a90ba8995c5591604135c325e0444a51"
Jul 27 18:32:02.541132 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:02.541079    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c1d1d60e94fe1f407091291f29c6f8a90ba8995c5591604135c325e0444a51"} err="failed to get container status \"31c1d1d60e94fe1f407091291f29c6f8a90ba8995c5591604135c325e0444a51\": rpc error: code = NotFound desc = could not find container \"31c1d1d60e94fe1f407091291f29c6f8a90ba8995c5591604135c325e0444a51\": container with ID starting with 31c1d1d60e94fe1f407091291f29c6f8a90ba8995c5591604135c325e0444a51 not found: ID does not exist"
Jul 27 18:32:02.549738 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:02.549716    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"]
Jul 27 18:32:02.557112 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:02.557078    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche7cxdm"]
Jul 27 18:32:04.399504 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:04.399455    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a47329b8-bd15-4554-bf44-c5f825292c41" path="/var/lib/kubelet/pods/a47329b8-bd15-4554-bf44-c5f825292c41/volumes"
Jul 27 18:32:36.357600 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.357559    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"]
Jul 27 18:32:36.358135 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.357938    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a47329b8-bd15-4554-bf44-c5f825292c41" containerName="main"
Jul 27 18:32:36.358135 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.357955    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a47329b8-bd15-4554-bf44-c5f825292c41" containerName="main"
Jul 27 18:32:36.358135 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.358045    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a47329b8-bd15-4554-bf44-c5f825292c41" containerName="main"
Jul 27 18:32:36.360590 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.360569    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.363351 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.363331    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 27 18:32:36.363476 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.363413    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-9xfm7\""
Jul 27 18:32:36.363561 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.363471    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 27 18:32:36.363561 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.363472    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 27 18:32:36.373782 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.373756    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"]
Jul 27 18:32:36.443422 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.443385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd309d91-1411-4b4b-bedb-3ae7c53a796b-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.443422 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.443425    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd309d91-1411-4b4b-bedb-3ae7c53a796b-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.443660 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.443553    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd309d91-1411-4b4b-bedb-3ae7c53a796b-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.443660 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.443589    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd309d91-1411-4b4b-bedb-3ae7c53a796b-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.443660 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.443617    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcb72\" (UniqueName: \"kubernetes.io/projected/fd309d91-1411-4b4b-bedb-3ae7c53a796b-kube-api-access-hcb72\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.443765 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.443661    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd309d91-1411-4b4b-bedb-3ae7c53a796b-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.543972 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.543926    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd309d91-1411-4b4b-bedb-3ae7c53a796b-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.544163 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.543976    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd309d91-1411-4b4b-bedb-3ae7c53a796b-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.544163 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.544019    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcb72\" (UniqueName: \"kubernetes.io/projected/fd309d91-1411-4b4b-bedb-3ae7c53a796b-kube-api-access-hcb72\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.544280 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.544155    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd309d91-1411-4b4b-bedb-3ae7c53a796b-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.544280 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.544240    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd309d91-1411-4b4b-bedb-3ae7c53a796b-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.544392 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.544280    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd309d91-1411-4b4b-bedb-3ae7c53a796b-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.544392 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.544365    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd309d91-1411-4b4b-bedb-3ae7c53a796b-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.544645 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.544620    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd309d91-1411-4b4b-bedb-3ae7c53a796b-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.544775 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.544628    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd309d91-1411-4b4b-bedb-3ae7c53a796b-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.546907 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.546886    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd309d91-1411-4b4b-bedb-3ae7c53a796b-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.547112 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.547090    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd309d91-1411-4b4b-bedb-3ae7c53a796b-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.558078 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.558049    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcb72\" (UniqueName: \"kubernetes.io/projected/fd309d91-1411-4b4b-bedb-3ae7c53a796b-kube-api-access-hcb72\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" (UID: \"fd309d91-1411-4b4b-bedb-3ae7c53a796b\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:32:36.578774 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.578744    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz"]
Jul 27 18:32:36.582227 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.582203    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz"
Jul 27 18:32:36.585071 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.585047    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-ldrms\""
Jul 27 18:32:36.593134 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.593111    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz"]
Jul 27 18:32:36.644991 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.644894    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-869b6\" (UniqueName: \"kubernetes.io/projected/4f32425c-6da8-436b-8640-ace532cc8c8d-kube-api-access-869b6\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz\" (UID: \"4f32425c-6da8-436b-8640-ace532cc8c8d\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz"
Jul 27 18:32:36.644991 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.644944    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f32425c-6da8-436b-8640-ace532cc8c8d-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz\" (UID: \"4f32425c-6da8-436b-8640-ace532cc8c8d\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz"
Jul 27 18:32:36.746117 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.746085    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-869b6\" (UniqueName: \"kubernetes.io/projected/4f32425c-6da8-436b-8640-ace532cc8c8d-kube-api-access-869b6\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz\" (UID: \"4f32425c-6da8-436b-8640-ace532cc8c8d\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz"
Jul 27 18:32:36.746117 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.746129    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f32425c-6da8-436b-8640-ace532cc8c8d-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz\" (UID: \"4f32425c-6da8-436b-8640-ace532cc8c8d\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz"
Jul 27 18:32:36.748625 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.748596    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f32425c-6da8-436b-8640-ace532cc8c8d-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz\" (UID: \"4f32425c-6da8-436b-8640-ace532cc8c8d\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz"
Jul 27 18:32:36.756981 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.756954    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-869b6\" (UniqueName: \"kubernetes.io/projected/4f32425c-6da8-436b-8640-ace532cc8c8d-kube-api-access-869b6\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz\" (UID: \"4f32425c-6da8-436b-8640-ace532cc8c8d\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz"
Jul 27 18:32:36.894773 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:36.894739    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz"
Jul 27 18:32:37.029527 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:37.029482    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz"]
Jul 27 18:32:37.031185 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:32:37.031157    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f32425c_6da8_436b_8640_ace532cc8c8d.slice/crio-260c176c56231afb8f7cbcaeb5976d1d278ab3ce917aed9070a046ba38771ff5 WatchSource:0}: Error finding container 260c176c56231afb8f7cbcaeb5976d1d278ab3ce917aed9070a046ba38771ff5: Status 404 returned error can't find the container with id 260c176c56231afb8f7cbcaeb5976d1d278ab3ce917aed9070a046ba38771ff5
Jul 27 18:32:37.706050 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:37.706009    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz" event={"ID":"4f32425c-6da8-436b-8640-ace532cc8c8d","Type":"ContainerStarted","Data":"fd0147235c3cc7112d7368b3a9101cf4e2d453946e463d8048d8fb94e9360c4a"}
Jul 27 18:32:37.706509 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:37.706071    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz" event={"ID":"4f32425c-6da8-436b-8640-ace532cc8c8d","Type":"ContainerStarted","Data":"260c176c56231afb8f7cbcaeb5976d1d278ab3ce917aed9070a046ba38771ff5"}
Jul 27 18:32:37.706509 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:37.706092    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz"
Jul 27 18:32:37.725634 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:32:37.725577    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz" podStartSLOduration=1.725561946 podStartE2EDuration="1.725561946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:32:37.723090577 +0000 UTC m=+3839.767828669" watchObservedRunningTime="2026-07-27 18:32:37.725561946 +0000 UTC m=+3839.770300026"
Jul 27 18:33:08.713022 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:33:08.712938    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz"
Jul 27 18:34:39.371789 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:34:39.371740    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 18:34:40.300780 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:34:40.300739    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:36:43.301771 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:36:43.301717    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 18:36:43.906009 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:36:43.905980    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:38:46.907000 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:38:46.906958    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 18:38:47.501056 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:38:47.501021    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:40:50.502143 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:40:50.502035    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 18:40:51.097424 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:40:51.097390    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:42:18.506720 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.506679    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 27 18:42:18.515459 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:42:18.515421    2571 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 18:42:18.515459 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:42:18.515459 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:42:18.515459 ip-10-0-131-74 kubenswrapper[2571]:  > containerID="0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1" cmd=["/bin/sleep","15"]
Jul 27 18:42:18.515739 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.515474    2571 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" message="" err=<
Jul 27 18:42:18.515739 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:42:18.515739 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:42:18.515739 ip-10-0-131-74 kubenswrapper[2571]:  >
Jul 27 18:42:18.515739 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:42:18.515485    2571 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 18:42:18.515739 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:42:18.515739 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:42:18.515739 ip-10-0-131-74 kubenswrapper[2571]:  > pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="cf7d1299-ccfb-439c-b411-06540586a0b8" containerName="main" containerID="cri-o://0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1"
Jul 27 18:42:18.515739 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.515534    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="cf7d1299-ccfb-439c-b411-06540586a0b8" containerName="main" containerID="cri-o://0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1" gracePeriod=60
Jul 27 18:42:18.766779 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.766705    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:42:18.866594 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.866545    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-model-cache\") pod \"cf7d1299-ccfb-439c-b411-06540586a0b8\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") "
Jul 27 18:42:18.866815 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.866615    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-tmp-dir\") pod \"cf7d1299-ccfb-439c-b411-06540586a0b8\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") "
Jul 27 18:42:18.866815 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.866652    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkcnf\" (UniqueName: \"kubernetes.io/projected/cf7d1299-ccfb-439c-b411-06540586a0b8-kube-api-access-wkcnf\") pod \"cf7d1299-ccfb-439c-b411-06540586a0b8\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") "
Jul 27 18:42:18.866815 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.866678    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-home\") pod \"cf7d1299-ccfb-439c-b411-06540586a0b8\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") "
Jul 27 18:42:18.866815 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.866724    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-dshm\") pod \"cf7d1299-ccfb-439c-b411-06540586a0b8\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") "
Jul 27 18:42:18.866815 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.866765    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7d1299-ccfb-439c-b411-06540586a0b8-tls-certs\") pod \"cf7d1299-ccfb-439c-b411-06540586a0b8\" (UID: \"cf7d1299-ccfb-439c-b411-06540586a0b8\") "
Jul 27 18:42:18.867116 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.866856    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf7d1299-ccfb-439c-b411-06540586a0b8" (UID: "cf7d1299-ccfb-439c-b411-06540586a0b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:42:18.867116 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.866877    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf7d1299-ccfb-439c-b411-06540586a0b8" (UID: "cf7d1299-ccfb-439c-b411-06540586a0b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:42:18.867116 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.867048    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:42:18.867116 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.867070    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:42:18.867116 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.867089    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-home" (OuterVolumeSpecName: "home") pod "cf7d1299-ccfb-439c-b411-06540586a0b8" (UID: "cf7d1299-ccfb-439c-b411-06540586a0b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:42:18.869685 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.869635    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7d1299-ccfb-439c-b411-06540586a0b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf7d1299-ccfb-439c-b411-06540586a0b8" (UID: "cf7d1299-ccfb-439c-b411-06540586a0b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:42:18.869685 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.869651    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7d1299-ccfb-439c-b411-06540586a0b8-kube-api-access-wkcnf" (OuterVolumeSpecName: "kube-api-access-wkcnf") pod "cf7d1299-ccfb-439c-b411-06540586a0b8" (UID: "cf7d1299-ccfb-439c-b411-06540586a0b8"). InnerVolumeSpecName "kube-api-access-wkcnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:42:18.869880 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.869772    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-dshm" (OuterVolumeSpecName: "dshm") pod "cf7d1299-ccfb-439c-b411-06540586a0b8" (UID: "cf7d1299-ccfb-439c-b411-06540586a0b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:42:18.967712 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.967674    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkcnf\" (UniqueName: \"kubernetes.io/projected/cf7d1299-ccfb-439c-b411-06540586a0b8-kube-api-access-wkcnf\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:42:18.967712 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.967705    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:42:18.967712 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.967716    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7d1299-ccfb-439c-b411-06540586a0b8-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:42:18.967969 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:18.967724    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7d1299-ccfb-439c-b411-06540586a0b8-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:42:19.536286 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:19.536251    2571 generic.go:356] "Generic (PLEG): container finished" podID="cf7d1299-ccfb-439c-b411-06540586a0b8" containerID="0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1" exitCode=0
Jul 27 18:42:19.536286 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:19.536291    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"cf7d1299-ccfb-439c-b411-06540586a0b8","Type":"ContainerDied","Data":"0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1"}
Jul 27 18:42:19.536918 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:19.536312    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 27 18:42:19.536918 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:19.536349    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"cf7d1299-ccfb-439c-b411-06540586a0b8","Type":"ContainerDied","Data":"7d3519892582ff55208f68f414b93bb0f22974902a1a15dd66ab3c426487d30f"}
Jul 27 18:42:19.536918 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:19.536381    2571 scope.go:117] "RemoveContainer" containerID="0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1"
Jul 27 18:42:19.546216 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:19.546200    2571 scope.go:117] "RemoveContainer" containerID="0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1"
Jul 27 18:42:19.546476 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:42:19.546454    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1\": container with ID starting with 0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1 not found: ID does not exist" containerID="0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1"
Jul 27 18:42:19.546560 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:19.546479    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1"} err="failed to get container status \"0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1\": rpc error: code = NotFound desc = could not find container \"0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1\": container with ID starting with 0f3949fdebf77f188f514bd960bcccd5fb864915bc33f17952a4bd8709d246c1 not found: ID does not exist"
Jul 27 18:42:19.558061 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:19.558034    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 27 18:42:19.563731 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:19.563707    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 27 18:42:20.399305 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:20.399268    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf7d1299-ccfb-439c-b411-06540586a0b8" path="/var/lib/kubelet/pods/cf7d1299-ccfb-439c-b411-06540586a0b8/volumes"
Jul 27 18:42:47.185130 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.185093    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 27 18:42:47.185544 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.185407    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf7d1299-ccfb-439c-b411-06540586a0b8" containerName="main"
Jul 27 18:42:47.185544 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.185418    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf7d1299-ccfb-439c-b411-06540586a0b8" containerName="main"
Jul 27 18:42:47.185544 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.185470    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf7d1299-ccfb-439c-b411-06540586a0b8" containerName="main"
Jul 27 18:42:47.189709 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.189690    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.193761 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.193731    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-mn-dockercfg-7dwrq\""
Jul 27 18:42:47.193908 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.193759    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 27 18:42:47.193908 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.193745    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 27 18:42:47.193908 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.193745    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 27 18:42:47.193908 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.193746    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-bcxpf\""
Jul 27 18:42:47.199386 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.199365    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 27 18:42:47.311427 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.311391    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.311629 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.311442    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk2cp\" (UniqueName: \"kubernetes.io/projected/f8ac8353-1de3-4ba1-a2a3-f333427bd285-kube-api-access-qk2cp\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.311629 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.311465    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.311629 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.311483    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.311629 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.311559    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.311629 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.311601    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ac8353-1de3-4ba1-a2a3-f333427bd285-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.412747 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.412710    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.412908 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.412760    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ac8353-1de3-4ba1-a2a3-f333427bd285-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.412908 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.412893    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.413033 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.412934    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2cp\" (UniqueName: \"kubernetes.io/projected/f8ac8353-1de3-4ba1-a2a3-f333427bd285-kube-api-access-qk2cp\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.413033 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.412956    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.413148 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.413064    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.413148 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.413086    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.413365 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.413329    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.413365 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.413336    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.414987 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.414971    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.415246 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.415230    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ac8353-1de3-4ba1-a2a3-f333427bd285-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.421256 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.421234    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2cp\" (UniqueName: \"kubernetes.io/projected/f8ac8353-1de3-4ba1-a2a3-f333427bd285-kube-api-access-qk2cp\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.500221 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.500184    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:42:47.637780 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.637738    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 27 18:42:47.640250 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:42:47.640219    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ac8353_1de3_4ba1_a2a3_f333427bd285.slice/crio-8430318c9f8ccdbdf066c8206f0265d449e4523e662fbc05372038129c37d969 WatchSource:0}: Error finding container 8430318c9f8ccdbdf066c8206f0265d449e4523e662fbc05372038129c37d969: Status 404 returned error can't find the container with id 8430318c9f8ccdbdf066c8206f0265d449e4523e662fbc05372038129c37d969
Jul 27 18:42:47.677971 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:47.677938    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"f8ac8353-1de3-4ba1-a2a3-f333427bd285","Type":"ContainerStarted","Data":"8430318c9f8ccdbdf066c8206f0265d449e4523e662fbc05372038129c37d969"}
Jul 27 18:42:48.683943 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:48.683907    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"f8ac8353-1de3-4ba1-a2a3-f333427bd285","Type":"ContainerStarted","Data":"38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd"}
Jul 27 18:42:48.702299 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:48.702244    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podStartSLOduration=1.702228792 podStartE2EDuration="1.702228792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:42:48.700331774 +0000 UTC m=+4450.745069857" watchObservedRunningTime="2026-07-27 18:42:48.702228792 +0000 UTC m=+4450.746966872"
Jul 27 18:42:54.098739 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:42:54.098686    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 18:42:54.712755 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:42:54.712723    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:44:57.713606 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:44:57.713556    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 18:44:58.312712 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:44:58.312676    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:47:01.313631 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:47:01.313533    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 18:47:01.916710 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:47:01.916672    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:47:28.126694 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:47:28.126663    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz_4f32425c-6da8-436b-8640-ace532cc8c8d/main/0.log"
Jul 27 18:48:24.166166 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.166128    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"]
Jul 27 18:48:24.169692 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.169668    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"
Jul 27 18:48:24.172628 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.172600    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 27 18:48:24.174010 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.173986    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 27 18:48:24.174135 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.173986    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-qbhqj\""
Jul 27 18:48:24.174135 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.174107    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 27 18:48:24.174256 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.174174    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-g5bb7\""
Jul 27 18:48:24.180942 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.180919    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"]
Jul 27 18:48:24.258289 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.258252    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbpp\" (UniqueName: \"kubernetes.io/projected/2f06c751-500e-48cf-9352-adfa61298192-kube-api-access-7xbpp\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj\" (UID: \"2f06c751-500e-48cf-9352-adfa61298192\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"
Jul 27 18:48:24.258489 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.258361    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f06c751-500e-48cf-9352-adfa61298192-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj\" (UID: \"2f06c751-500e-48cf-9352-adfa61298192\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"
Jul 27 18:48:24.359105 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.359064    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbpp\" (UniqueName: \"kubernetes.io/projected/2f06c751-500e-48cf-9352-adfa61298192-kube-api-access-7xbpp\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj\" (UID: \"2f06c751-500e-48cf-9352-adfa61298192\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"
Jul 27 18:48:24.359286 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.359131    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f06c751-500e-48cf-9352-adfa61298192-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj\" (UID: \"2f06c751-500e-48cf-9352-adfa61298192\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"
Jul 27 18:48:24.361686 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.361661    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f06c751-500e-48cf-9352-adfa61298192-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj\" (UID: \"2f06c751-500e-48cf-9352-adfa61298192\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"
Jul 27 18:48:24.373125 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.373096    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbpp\" (UniqueName: \"kubernetes.io/projected/2f06c751-500e-48cf-9352-adfa61298192-kube-api-access-7xbpp\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj\" (UID: \"2f06c751-500e-48cf-9352-adfa61298192\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"
Jul 27 18:48:24.480910 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.480869    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"
Jul 27 18:48:24.613911 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:24.613846    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"]
Jul 27 18:48:24.616328 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:48:24.616299    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f06c751_500e_48cf_9352_adfa61298192.slice/crio-ff64328602fb81d255abb99e06166e16c98306d32521b46f78a516163e4057dc WatchSource:0}: Error finding container ff64328602fb81d255abb99e06166e16c98306d32521b46f78a516163e4057dc: Status 404 returned error can't find the container with id ff64328602fb81d255abb99e06166e16c98306d32521b46f78a516163e4057dc
Jul 27 18:48:25.328143 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:25.328094    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj" event={"ID":"2f06c751-500e-48cf-9352-adfa61298192","Type":"ContainerStarted","Data":"14cea64fd96009fff460d9d0a672b87db6f72fe8e5d23ef728a87443be4b83eb"}
Jul 27 18:48:25.328703 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:25.328299    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"
Jul 27 18:48:25.328703 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:25.328339    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj" event={"ID":"2f06c751-500e-48cf-9352-adfa61298192","Type":"ContainerStarted","Data":"ff64328602fb81d255abb99e06166e16c98306d32521b46f78a516163e4057dc"}
Jul 27 18:48:25.350364 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:25.350315    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj" podStartSLOduration=1.350300963 podStartE2EDuration="1.350300963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:48:25.3486488 +0000 UTC m=+4787.393386897" watchObservedRunningTime="2026-07-27 18:48:25.350300963 +0000 UTC m=+4787.395039044"
Jul 27 18:48:56.335322 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:48:56.335288    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"
Jul 27 18:49:04.917911 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:49:04.917862    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 18:49:05.527651 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:49:05.527618    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:50:31.735551 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:31.735511    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"]
Jul 27 18:50:36.334886 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:36.334843    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj" podUID="2f06c751-500e-48cf-9352-adfa61298192" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:50:46.335219 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:46.335178    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj" podUID="2f06c751-500e-48cf-9352-adfa61298192" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:50:46.754630 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:46.754587    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj" podUID="2f06c751-500e-48cf-9352-adfa61298192" containerName="main" containerID="cri-o://14cea64fd96009fff460d9d0a672b87db6f72fe8e5d23ef728a87443be4b83eb" gracePeriod=45
Jul 27 18:50:46.996082 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:46.996052    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"
Jul 27 18:50:47.033730 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.033650    2571 generic.go:356] "Generic (PLEG): container finished" podID="2f06c751-500e-48cf-9352-adfa61298192" containerID="14cea64fd96009fff460d9d0a672b87db6f72fe8e5d23ef728a87443be4b83eb" exitCode=0
Jul 27 18:50:47.033730 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.033690    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj" event={"ID":"2f06c751-500e-48cf-9352-adfa61298192","Type":"ContainerDied","Data":"14cea64fd96009fff460d9d0a672b87db6f72fe8e5d23ef728a87443be4b83eb"}
Jul 27 18:50:47.033943 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.033730    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj" event={"ID":"2f06c751-500e-48cf-9352-adfa61298192","Type":"ContainerDied","Data":"ff64328602fb81d255abb99e06166e16c98306d32521b46f78a516163e4057dc"}
Jul 27 18:50:47.033943 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.033753    2571 scope.go:117] "RemoveContainer" containerID="14cea64fd96009fff460d9d0a672b87db6f72fe8e5d23ef728a87443be4b83eb"
Jul 27 18:50:47.033943 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.033692    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"
Jul 27 18:50:47.042368 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.042343    2571 scope.go:117] "RemoveContainer" containerID="14cea64fd96009fff460d9d0a672b87db6f72fe8e5d23ef728a87443be4b83eb"
Jul 27 18:50:47.042660 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:50:47.042640    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14cea64fd96009fff460d9d0a672b87db6f72fe8e5d23ef728a87443be4b83eb\": container with ID starting with 14cea64fd96009fff460d9d0a672b87db6f72fe8e5d23ef728a87443be4b83eb not found: ID does not exist" containerID="14cea64fd96009fff460d9d0a672b87db6f72fe8e5d23ef728a87443be4b83eb"
Jul 27 18:50:47.042746 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.042668    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14cea64fd96009fff460d9d0a672b87db6f72fe8e5d23ef728a87443be4b83eb"} err="failed to get container status \"14cea64fd96009fff460d9d0a672b87db6f72fe8e5d23ef728a87443be4b83eb\": rpc error: code = NotFound desc = could not find container \"14cea64fd96009fff460d9d0a672b87db6f72fe8e5d23ef728a87443be4b83eb\": container with ID starting with 14cea64fd96009fff460d9d0a672b87db6f72fe8e5d23ef728a87443be4b83eb not found: ID does not exist"
Jul 27 18:50:47.085694 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.085660    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f06c751-500e-48cf-9352-adfa61298192-tls-certs\") pod \"2f06c751-500e-48cf-9352-adfa61298192\" (UID: \"2f06c751-500e-48cf-9352-adfa61298192\") "
Jul 27 18:50:47.085853 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.085754    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xbpp\" (UniqueName: \"kubernetes.io/projected/2f06c751-500e-48cf-9352-adfa61298192-kube-api-access-7xbpp\") pod \"2f06c751-500e-48cf-9352-adfa61298192\" (UID: \"2f06c751-500e-48cf-9352-adfa61298192\") "
Jul 27 18:50:47.087873 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.087841    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f06c751-500e-48cf-9352-adfa61298192-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f06c751-500e-48cf-9352-adfa61298192" (UID: "2f06c751-500e-48cf-9352-adfa61298192"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:50:47.087991 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.087897    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f06c751-500e-48cf-9352-adfa61298192-kube-api-access-7xbpp" (OuterVolumeSpecName: "kube-api-access-7xbpp") pod "2f06c751-500e-48cf-9352-adfa61298192" (UID: "2f06c751-500e-48cf-9352-adfa61298192"). InnerVolumeSpecName "kube-api-access-7xbpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:50:47.187016 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.186974    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xbpp\" (UniqueName: \"kubernetes.io/projected/2f06c751-500e-48cf-9352-adfa61298192-kube-api-access-7xbpp\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:50:47.187016 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.187008    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f06c751-500e-48cf-9352-adfa61298192-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:50:47.356346 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.356310    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"]
Jul 27 18:50:47.360132 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:47.360106    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schem66xj"]
Jul 27 18:50:48.398476 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:50:48.398434    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f06c751-500e-48cf-9352-adfa61298192" path="/var/lib/kubelet/pods/2f06c751-500e-48cf-9352-adfa61298192/volumes"
Jul 27 18:51:08.528641 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:51:08.528538    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 18:51:09.147588 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.147550    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:51:09.651170 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.651136    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"]
Jul 27 18:51:09.651675 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.651612    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f06c751-500e-48cf-9352-adfa61298192" containerName="main"
Jul 27 18:51:09.651675 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.651632    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f06c751-500e-48cf-9352-adfa61298192" containerName="main"
Jul 27 18:51:09.651796 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.651773    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f06c751-500e-48cf-9352-adfa61298192" containerName="main"
Jul 27 18:51:09.654670 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.654650    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"
Jul 27 18:51:09.657372 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.657349    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Jul 27 18:51:09.658827 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.658754    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Jul 27 18:51:09.658945 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.658931    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-n4grp\""
Jul 27 18:51:09.659017 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.658949    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-lswt7\""
Jul 27 18:51:09.659233 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.659131    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Jul 27 18:51:09.662804 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.662781    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"]
Jul 27 18:51:09.787353 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.787310    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023a8976-b852-4fb0-98f7-9f1178b01826-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz\" (UID: \"023a8976-b852-4fb0-98f7-9f1178b01826\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"
Jul 27 18:51:09.787353 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.787353    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz6lm\" (UniqueName: \"kubernetes.io/projected/023a8976-b852-4fb0-98f7-9f1178b01826-kube-api-access-xz6lm\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz\" (UID: \"023a8976-b852-4fb0-98f7-9f1178b01826\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"
Jul 27 18:51:09.888461 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.888417    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023a8976-b852-4fb0-98f7-9f1178b01826-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz\" (UID: \"023a8976-b852-4fb0-98f7-9f1178b01826\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"
Jul 27 18:51:09.888686 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.888468    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz6lm\" (UniqueName: \"kubernetes.io/projected/023a8976-b852-4fb0-98f7-9f1178b01826-kube-api-access-xz6lm\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz\" (UID: \"023a8976-b852-4fb0-98f7-9f1178b01826\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"
Jul 27 18:51:09.891152 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.891128    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023a8976-b852-4fb0-98f7-9f1178b01826-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz\" (UID: \"023a8976-b852-4fb0-98f7-9f1178b01826\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"
Jul 27 18:51:09.896644 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.896612    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz6lm\" (UniqueName: \"kubernetes.io/projected/023a8976-b852-4fb0-98f7-9f1178b01826-kube-api-access-xz6lm\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz\" (UID: \"023a8976-b852-4fb0-98f7-9f1178b01826\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"
Jul 27 18:51:09.965415 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:09.965381    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"
Jul 27 18:51:10.102682 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:10.102657    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"]
Jul 27 18:51:10.105149 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:51:10.105119    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023a8976_b852_4fb0_98f7_9f1178b01826.slice/crio-a291381c9572afce77dd61cf66ef8aa355d8348e4d6fdbd8530de084ac923712 WatchSource:0}: Error finding container a291381c9572afce77dd61cf66ef8aa355d8348e4d6fdbd8530de084ac923712: Status 404 returned error can't find the container with id a291381c9572afce77dd61cf66ef8aa355d8348e4d6fdbd8530de084ac923712
Jul 27 18:51:10.154633 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:10.154597    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz" event={"ID":"023a8976-b852-4fb0-98f7-9f1178b01826","Type":"ContainerStarted","Data":"a291381c9572afce77dd61cf66ef8aa355d8348e4d6fdbd8530de084ac923712"}
Jul 27 18:51:11.161447 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:11.161403    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz" event={"ID":"023a8976-b852-4fb0-98f7-9f1178b01826","Type":"ContainerStarted","Data":"41c9eabe06c6f78e8c2a3fdf0c88761bc42802d89f5048d2fd7af11aa2fd49c8"}
Jul 27 18:51:11.161964 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:11.161519    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"
Jul 27 18:51:11.179361 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:11.179303    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz" podStartSLOduration=2.179288028 podStartE2EDuration="2.179288028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:51:11.177097544 +0000 UTC m=+4953.221835627" watchObservedRunningTime="2026-07-27 18:51:11.179288028 +0000 UTC m=+4953.224026109"
Jul 27 18:51:19.105084 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:19.105047    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"]
Jul 27 18:51:34.124881 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:34.124814    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz" podUID="023a8976-b852-4fb0-98f7-9f1178b01826" containerName="main" containerID="cri-o://41c9eabe06c6f78e8c2a3fdf0c88761bc42802d89f5048d2fd7af11aa2fd49c8" gracePeriod=45
Jul 27 18:51:34.283824 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:34.283790    2571 generic.go:356] "Generic (PLEG): container finished" podID="023a8976-b852-4fb0-98f7-9f1178b01826" containerID="41c9eabe06c6f78e8c2a3fdf0c88761bc42802d89f5048d2fd7af11aa2fd49c8" exitCode=0
Jul 27 18:51:34.284005 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:34.283864    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz" event={"ID":"023a8976-b852-4fb0-98f7-9f1178b01826","Type":"ContainerDied","Data":"41c9eabe06c6f78e8c2a3fdf0c88761bc42802d89f5048d2fd7af11aa2fd49c8"}
Jul 27 18:51:34.379791 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:34.379762    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"
Jul 27 18:51:34.514202 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:34.514098    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023a8976-b852-4fb0-98f7-9f1178b01826-tls-certs\") pod \"023a8976-b852-4fb0-98f7-9f1178b01826\" (UID: \"023a8976-b852-4fb0-98f7-9f1178b01826\") "
Jul 27 18:51:34.514202 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:34.514151    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz6lm\" (UniqueName: \"kubernetes.io/projected/023a8976-b852-4fb0-98f7-9f1178b01826-kube-api-access-xz6lm\") pod \"023a8976-b852-4fb0-98f7-9f1178b01826\" (UID: \"023a8976-b852-4fb0-98f7-9f1178b01826\") "
Jul 27 18:51:34.516393 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:34.516361    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023a8976-b852-4fb0-98f7-9f1178b01826-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "023a8976-b852-4fb0-98f7-9f1178b01826" (UID: "023a8976-b852-4fb0-98f7-9f1178b01826"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:51:34.516524 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:34.516428    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023a8976-b852-4fb0-98f7-9f1178b01826-kube-api-access-xz6lm" (OuterVolumeSpecName: "kube-api-access-xz6lm") pod "023a8976-b852-4fb0-98f7-9f1178b01826" (UID: "023a8976-b852-4fb0-98f7-9f1178b01826"). InnerVolumeSpecName "kube-api-access-xz6lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:51:34.615052 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:34.615010    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023a8976-b852-4fb0-98f7-9f1178b01826-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:51:34.615052 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:34.615046    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz6lm\" (UniqueName: \"kubernetes.io/projected/023a8976-b852-4fb0-98f7-9f1178b01826-kube-api-access-xz6lm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:51:35.290255 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:35.290225    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"
Jul 27 18:51:35.290754 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:35.290253    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz" event={"ID":"023a8976-b852-4fb0-98f7-9f1178b01826","Type":"ContainerDied","Data":"a291381c9572afce77dd61cf66ef8aa355d8348e4d6fdbd8530de084ac923712"}
Jul 27 18:51:35.290754 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:35.290316    2571 scope.go:117] "RemoveContainer" containerID="41c9eabe06c6f78e8c2a3fdf0c88761bc42802d89f5048d2fd7af11aa2fd49c8"
Jul 27 18:51:35.313062 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:35.313034    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"]
Jul 27 18:51:35.316612 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:35.316586    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-wpvhz"]
Jul 27 18:51:36.399391 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:51:36.399350    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="023a8976-b852-4fb0-98f7-9f1178b01826" path="/var/lib/kubelet/pods/023a8976-b852-4fb0-98f7-9f1178b01826/volumes"
Jul 27 18:52:08.014128 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.014081    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"]
Jul 27 18:52:08.014768 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.014572    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023a8976-b852-4fb0-98f7-9f1178b01826" containerName="main"
Jul 27 18:52:08.014768 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.014594    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="023a8976-b852-4fb0-98f7-9f1178b01826" containerName="main"
Jul 27 18:52:08.014768 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.014695    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="023a8976-b852-4fb0-98f7-9f1178b01826" containerName="main"
Jul 27 18:52:08.017790 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.017768    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.020747 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.020724    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 27 18:52:08.020892 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.020821    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-wt2gj\""
Jul 27 18:52:08.020960 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.020888    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 27 18:52:08.020960 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.020907    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 27 18:52:08.021075 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.020981    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 27 18:52:08.028999 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.028978    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"]
Jul 27 18:52:08.096051 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.096017    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aeeb804-68d1-44c5-bd99-03fcae500228-tls-certs\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.096051 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.096054    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-home\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.096291 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.096076    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-tmp-dir\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.096291 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.096117    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-dshm\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.096291 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.096136    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5mps\" (UniqueName: \"kubernetes.io/projected/5aeeb804-68d1-44c5-bd99-03fcae500228-kube-api-access-q5mps\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.096291 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.096171    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-kserve-provision-location\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.096291 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.096194    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-model-cache\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.196688 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.196652    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-dshm\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.196688 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.196691    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5mps\" (UniqueName: \"kubernetes.io/projected/5aeeb804-68d1-44c5-bd99-03fcae500228-kube-api-access-q5mps\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.197016 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.196719    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-kserve-provision-location\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.197016 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.196742    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-model-cache\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.197016 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.196814    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aeeb804-68d1-44c5-bd99-03fcae500228-tls-certs\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.197016 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.196840    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-home\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.197016 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.196861    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-tmp-dir\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.197313 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.197263    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-kserve-provision-location\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.197359 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.197310    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-tmp-dir\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.197622 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.197603    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-model-cache\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.197690 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.197645    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-home\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.199374 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.199355    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-dshm\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.199717 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.199698    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aeeb804-68d1-44c5-bd99-03fcae500228-tls-certs\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.205775 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.205754    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5mps\" (UniqueName: \"kubernetes.io/projected/5aeeb804-68d1-44c5-bd99-03fcae500228-kube-api-access-q5mps\") pod \"stop-feature-test-kserve-57d79c4c49-ljrqf\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.328705 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.328616    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:08.467273 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:08.467248    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"]
Jul 27 18:52:08.469428 ip-10-0-131-74 kubenswrapper[2571]: W0727 18:52:08.469391    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aeeb804_68d1_44c5_bd99_03fcae500228.slice/crio-be342454a742a788e502a12ca6e6bec83b061b2bb09f9d73ebeb3bd335a53194 WatchSource:0}: Error finding container be342454a742a788e502a12ca6e6bec83b061b2bb09f9d73ebeb3bd335a53194: Status 404 returned error can't find the container with id be342454a742a788e502a12ca6e6bec83b061b2bb09f9d73ebeb3bd335a53194
Jul 27 18:52:09.456917 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:09.456876    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" event={"ID":"5aeeb804-68d1-44c5-bd99-03fcae500228","Type":"ContainerStarted","Data":"afd0af94083d5b4abeadad489e9b1173cb4247bd85cbf379918bd886a5cc547e"}
Jul 27 18:52:09.457390 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:09.456964    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" event={"ID":"5aeeb804-68d1-44c5-bd99-03fcae500228","Type":"ContainerStarted","Data":"be342454a742a788e502a12ca6e6bec83b061b2bb09f9d73ebeb3bd335a53194"}
Jul 27 18:52:16.498281 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:16.498242    2571 generic.go:356] "Generic (PLEG): container finished" podID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerID="afd0af94083d5b4abeadad489e9b1173cb4247bd85cbf379918bd886a5cc547e" exitCode=0
Jul 27 18:52:16.498763 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:16.498313    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" event={"ID":"5aeeb804-68d1-44c5-bd99-03fcae500228","Type":"ContainerDied","Data":"afd0af94083d5b4abeadad489e9b1173cb4247bd85cbf379918bd886a5cc547e"}
Jul 27 18:52:17.505992 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:17.505960    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" event={"ID":"5aeeb804-68d1-44c5-bd99-03fcae500228","Type":"ContainerStarted","Data":"0fc123fd1065b378ce6ed7fcb901d2e4aeb9dc8eac6ac67636c5a921b86580a0"}
Jul 27 18:52:17.527323 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:17.527271    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podStartSLOduration=3.023974852 podStartE2EDuration="10.527255935s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.503281083s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:52:17.525900481 +0000 UTC m=+5019.570638576" watchObservedRunningTime="2026-07-27 18:52:17.527255935 +0000 UTC m=+5019.571994017"
Jul 27 18:52:18.329169 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:18.329128    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:18.329359 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:18.329204    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:52:18.330889 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:18.330853    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 27 18:52:28.329918 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:28.329867    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 27 18:52:38.329088 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:38.329039    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 27 18:52:48.329990 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:48.329941    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 27 18:52:58.329691 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:52:58.329652    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 27 18:53:08.329695 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:53:08.329647    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 27 18:53:12.149003 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:53:12.148954    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 18:53:12.790566 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:53:12.790532    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:53:18.329857 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:53:18.329807    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 27 18:53:28.329668 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:53:28.329620    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 27 18:53:38.329963 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:53:38.329915    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 27 18:53:48.329711 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:53:48.329661    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 27 18:53:58.329089 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:53:58.329041    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 27 18:54:08.339322 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:08.339231    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:54:08.347004 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:08.346975    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:54:15.981897 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:15.981862    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"]
Jul 27 18:54:31.004586 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.004536    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main" containerID="cri-o://0fc123fd1065b378ce6ed7fcb901d2e4aeb9dc8eac6ac67636c5a921b86580a0" gracePeriod=45
Jul 27 18:54:31.340222 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.340122    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 27 18:54:31.679677 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.679651    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:54:31.823275 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.823233    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-dshm\") pod \"5aeeb804-68d1-44c5-bd99-03fcae500228\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") "
Jul 27 18:54:31.823275 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.823280    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5mps\" (UniqueName: \"kubernetes.io/projected/5aeeb804-68d1-44c5-bd99-03fcae500228-kube-api-access-q5mps\") pod \"5aeeb804-68d1-44c5-bd99-03fcae500228\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") "
Jul 27 18:54:31.823524 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.823324    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aeeb804-68d1-44c5-bd99-03fcae500228-tls-certs\") pod \"5aeeb804-68d1-44c5-bd99-03fcae500228\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") "
Jul 27 18:54:31.823524 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.823383    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-model-cache\") pod \"5aeeb804-68d1-44c5-bd99-03fcae500228\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") "
Jul 27 18:54:31.823524 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.823431    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-home\") pod \"5aeeb804-68d1-44c5-bd99-03fcae500228\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") "
Jul 27 18:54:31.823524 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.823509    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-kserve-provision-location\") pod \"5aeeb804-68d1-44c5-bd99-03fcae500228\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") "
Jul 27 18:54:31.823751 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.823539    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-tmp-dir\") pod \"5aeeb804-68d1-44c5-bd99-03fcae500228\" (UID: \"5aeeb804-68d1-44c5-bd99-03fcae500228\") "
Jul 27 18:54:31.823809 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.823707    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-model-cache" (OuterVolumeSpecName: "model-cache") pod "5aeeb804-68d1-44c5-bd99-03fcae500228" (UID: "5aeeb804-68d1-44c5-bd99-03fcae500228"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:54:31.823883 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.823864    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:54:31.824193 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.824169    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-home" (OuterVolumeSpecName: "home") pod "5aeeb804-68d1-44c5-bd99-03fcae500228" (UID: "5aeeb804-68d1-44c5-bd99-03fcae500228"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:54:31.825676 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.825648    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aeeb804-68d1-44c5-bd99-03fcae500228-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5aeeb804-68d1-44c5-bd99-03fcae500228" (UID: "5aeeb804-68d1-44c5-bd99-03fcae500228"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:54:31.825872 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.825850    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aeeb804-68d1-44c5-bd99-03fcae500228-kube-api-access-q5mps" (OuterVolumeSpecName: "kube-api-access-q5mps") pod "5aeeb804-68d1-44c5-bd99-03fcae500228" (UID: "5aeeb804-68d1-44c5-bd99-03fcae500228"). InnerVolumeSpecName "kube-api-access-q5mps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:54:31.826133 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.826108    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-dshm" (OuterVolumeSpecName: "dshm") pod "5aeeb804-68d1-44c5-bd99-03fcae500228" (UID: "5aeeb804-68d1-44c5-bd99-03fcae500228"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:54:31.836218 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.836175    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5aeeb804-68d1-44c5-bd99-03fcae500228" (UID: "5aeeb804-68d1-44c5-bd99-03fcae500228"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:54:31.884099 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.884010    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5aeeb804-68d1-44c5-bd99-03fcae500228" (UID: "5aeeb804-68d1-44c5-bd99-03fcae500228"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:54:31.924425 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.924385    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:54:31.924425 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.924421    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-kserve-provision-location\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:54:31.924425 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.924430    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:54:31.924698 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.924440    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aeeb804-68d1-44c5-bd99-03fcae500228-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:54:31.924698 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.924449    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5mps\" (UniqueName: \"kubernetes.io/projected/5aeeb804-68d1-44c5-bd99-03fcae500228-kube-api-access-q5mps\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:54:31.924698 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:31.924457    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aeeb804-68d1-44c5-bd99-03fcae500228-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:54:32.195068 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:32.194974    2571 generic.go:356] "Generic (PLEG): container finished" podID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerID="0fc123fd1065b378ce6ed7fcb901d2e4aeb9dc8eac6ac67636c5a921b86580a0" exitCode=0
Jul 27 18:54:32.195068 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:32.195026    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"
Jul 27 18:54:32.195633 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:32.195032    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" event={"ID":"5aeeb804-68d1-44c5-bd99-03fcae500228","Type":"ContainerDied","Data":"0fc123fd1065b378ce6ed7fcb901d2e4aeb9dc8eac6ac67636c5a921b86580a0"}
Jul 27 18:54:32.195633 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:32.195150    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf" event={"ID":"5aeeb804-68d1-44c5-bd99-03fcae500228","Type":"ContainerDied","Data":"be342454a742a788e502a12ca6e6bec83b061b2bb09f9d73ebeb3bd335a53194"}
Jul 27 18:54:32.195633 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:32.195184    2571 scope.go:117] "RemoveContainer" containerID="0fc123fd1065b378ce6ed7fcb901d2e4aeb9dc8eac6ac67636c5a921b86580a0"
Jul 27 18:54:32.205301 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:32.205279    2571 scope.go:117] "RemoveContainer" containerID="afd0af94083d5b4abeadad489e9b1173cb4247bd85cbf379918bd886a5cc547e"
Jul 27 18:54:32.213817 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:32.213795    2571 scope.go:117] "RemoveContainer" containerID="0fc123fd1065b378ce6ed7fcb901d2e4aeb9dc8eac6ac67636c5a921b86580a0"
Jul 27 18:54:32.214119 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:54:32.214097    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc123fd1065b378ce6ed7fcb901d2e4aeb9dc8eac6ac67636c5a921b86580a0\": container with ID starting with 0fc123fd1065b378ce6ed7fcb901d2e4aeb9dc8eac6ac67636c5a921b86580a0 not found: ID does not exist" containerID="0fc123fd1065b378ce6ed7fcb901d2e4aeb9dc8eac6ac67636c5a921b86580a0"
Jul 27 18:54:32.214166 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:32.214128    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc123fd1065b378ce6ed7fcb901d2e4aeb9dc8eac6ac67636c5a921b86580a0"} err="failed to get container status \"0fc123fd1065b378ce6ed7fcb901d2e4aeb9dc8eac6ac67636c5a921b86580a0\": rpc error: code = NotFound desc = could not find container \"0fc123fd1065b378ce6ed7fcb901d2e4aeb9dc8eac6ac67636c5a921b86580a0\": container with ID starting with 0fc123fd1065b378ce6ed7fcb901d2e4aeb9dc8eac6ac67636c5a921b86580a0 not found: ID does not exist"
Jul 27 18:54:32.214166 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:32.214148    2571 scope.go:117] "RemoveContainer" containerID="afd0af94083d5b4abeadad489e9b1173cb4247bd85cbf379918bd886a5cc547e"
Jul 27 18:54:32.214403 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:54:32.214382    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd0af94083d5b4abeadad489e9b1173cb4247bd85cbf379918bd886a5cc547e\": container with ID starting with afd0af94083d5b4abeadad489e9b1173cb4247bd85cbf379918bd886a5cc547e not found: ID does not exist" containerID="afd0af94083d5b4abeadad489e9b1173cb4247bd85cbf379918bd886a5cc547e"
Jul 27 18:54:32.214447 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:32.214409    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd0af94083d5b4abeadad489e9b1173cb4247bd85cbf379918bd886a5cc547e"} err="failed to get container status \"afd0af94083d5b4abeadad489e9b1173cb4247bd85cbf379918bd886a5cc547e\": rpc error: code = NotFound desc = could not find container \"afd0af94083d5b4abeadad489e9b1173cb4247bd85cbf379918bd886a5cc547e\": container with ID starting with afd0af94083d5b4abeadad489e9b1173cb4247bd85cbf379918bd886a5cc547e not found: ID does not exist"
Jul 27 18:54:32.218350 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:32.218324    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"]
Jul 27 18:54:32.224132 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:32.224108    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-ljrqf"]
Jul 27 18:54:32.399950 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:54:32.399914    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" path="/var/lib/kubelet/pods/5aeeb804-68d1-44c5-bd99-03fcae500228/volumes"
Jul 27 18:55:15.791445 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:55:15.791405    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 18:55:16.416162 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:55:16.416127    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:57:19.417030 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:57:19.416987    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 18:57:20.008891 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:20.008854    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 18:57:54.217303 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.217262    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 27 18:57:54.226410 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:57:54.226375    2571 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 18:57:54.226410 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:57:54.226410 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:57:54.226410 ip-10-0-131-74 kubenswrapper[2571]:  > containerID="38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd" cmd=["/bin/sleep","15"]
Jul 27 18:57:54.226673 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.226425    2571 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" message="" err=<
Jul 27 18:57:54.226673 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:57:54.226673 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:57:54.226673 ip-10-0-131-74 kubenswrapper[2571]:  >
Jul 27 18:57:54.226673 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:57:54.226437    2571 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 18:57:54.226673 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:57:54.226673 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 18:57:54.226673 ip-10-0-131-74 kubenswrapper[2571]:  > pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="f8ac8353-1de3-4ba1-a2a3-f333427bd285" containerName="main" containerID="cri-o://38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd"
Jul 27 18:57:54.226673 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.226474    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="f8ac8353-1de3-4ba1-a2a3-f333427bd285" containerName="main" containerID="cri-o://38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd" gracePeriod=60
Jul 27 18:57:54.594918 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.594894    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:57:54.663431 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.663400    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-home\") pod \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") "
Jul 27 18:57:54.663629 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.663445    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-model-cache\") pod \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") "
Jul 27 18:57:54.663629 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.663475    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ac8353-1de3-4ba1-a2a3-f333427bd285-tls-certs\") pod \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") "
Jul 27 18:57:54.663629 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.663551    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-dshm\") pod \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") "
Jul 27 18:57:54.663791 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.663693    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk2cp\" (UniqueName: \"kubernetes.io/projected/f8ac8353-1de3-4ba1-a2a3-f333427bd285-kube-api-access-qk2cp\") pod \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") "
Jul 27 18:57:54.663791 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.663748    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-home" (OuterVolumeSpecName: "home") pod "f8ac8353-1de3-4ba1-a2a3-f333427bd285" (UID: "f8ac8353-1de3-4ba1-a2a3-f333427bd285"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:57:54.663791 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.663750    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-tmp-dir\") pod \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\" (UID: \"f8ac8353-1de3-4ba1-a2a3-f333427bd285\") "
Jul 27 18:57:54.663791 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.663778    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8ac8353-1de3-4ba1-a2a3-f333427bd285" (UID: "f8ac8353-1de3-4ba1-a2a3-f333427bd285"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:57:54.664015 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.663980    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8ac8353-1de3-4ba1-a2a3-f333427bd285" (UID: "f8ac8353-1de3-4ba1-a2a3-f333427bd285"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:57:54.664140 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.664122    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:57:54.664192 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.664145    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:57:54.664192 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.664156    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:57:54.665898 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.665868    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ac8353-1de3-4ba1-a2a3-f333427bd285-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8ac8353-1de3-4ba1-a2a3-f333427bd285" (UID: "f8ac8353-1de3-4ba1-a2a3-f333427bd285"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:57:54.666005 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.665909    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-dshm" (OuterVolumeSpecName: "dshm") pod "f8ac8353-1de3-4ba1-a2a3-f333427bd285" (UID: "f8ac8353-1de3-4ba1-a2a3-f333427bd285"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:57:54.666005 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.665932    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ac8353-1de3-4ba1-a2a3-f333427bd285-kube-api-access-qk2cp" (OuterVolumeSpecName: "kube-api-access-qk2cp") pod "f8ac8353-1de3-4ba1-a2a3-f333427bd285" (UID: "f8ac8353-1de3-4ba1-a2a3-f333427bd285"). InnerVolumeSpecName "kube-api-access-qk2cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:57:54.764943 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.764846    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk2cp\" (UniqueName: \"kubernetes.io/projected/f8ac8353-1de3-4ba1-a2a3-f333427bd285-kube-api-access-qk2cp\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:57:54.764943 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.764881    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ac8353-1de3-4ba1-a2a3-f333427bd285-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:57:54.764943 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:54.764896    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ac8353-1de3-4ba1-a2a3-f333427bd285-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 18:57:55.188765 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:55.188662    2571 generic.go:356] "Generic (PLEG): container finished" podID="f8ac8353-1de3-4ba1-a2a3-f333427bd285" containerID="38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd" exitCode=0
Jul 27 18:57:55.188765 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:55.188710    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:57:55.188969 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:55.188757    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"f8ac8353-1de3-4ba1-a2a3-f333427bd285","Type":"ContainerDied","Data":"38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd"}
Jul 27 18:57:55.188969 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:55.188815    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"f8ac8353-1de3-4ba1-a2a3-f333427bd285","Type":"ContainerDied","Data":"8430318c9f8ccdbdf066c8206f0265d449e4523e662fbc05372038129c37d969"}
Jul 27 18:57:55.188969 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:55.188838    2571 scope.go:117] "RemoveContainer" containerID="38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd"
Jul 27 18:57:55.198839 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:55.198822    2571 scope.go:117] "RemoveContainer" containerID="38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd"
Jul 27 18:57:55.199105 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:57:55.199089    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd\": container with ID starting with 38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd not found: ID does not exist" containerID="38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd"
Jul 27 18:57:55.199147 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:55.199114    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd"} err="failed to get container status \"38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd\": rpc error: code = NotFound desc = could not find container \"38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd\": container with ID starting with 38785dccb12334cf0a19b48cce21f76eb0195a8b1f336ce13611446e94bdfcdd not found: ID does not exist"
Jul 27 18:57:55.210581 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:55.210557    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 27 18:57:55.212698 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:55.212678    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 27 18:57:56.398773 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:57:56.398734    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8ac8353-1de3-4ba1-a2a3-f333427bd285" path="/var/lib/kubelet/pods/f8ac8353-1de3-4ba1-a2a3-f333427bd285/volumes"
Jul 27 18:59:23.010413 ip-10-0-131-74 kubenswrapper[2571]: E0727 18:59:23.010362    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 18:59:23.655704 ip-10-0-131-74 kubenswrapper[2571]: I0727 18:59:23.655669    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 19:00:21.989638 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.989594    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"]
Jul 27 19:00:21.990187 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.990037    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main"
Jul 27 19:00:21.990187 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.990055    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main"
Jul 27 19:00:21.990187 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.990080    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ac8353-1de3-4ba1-a2a3-f333427bd285" containerName="main"
Jul 27 19:00:21.990187 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.990089    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ac8353-1de3-4ba1-a2a3-f333427bd285" containerName="main"
Jul 27 19:00:21.990187 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.990105    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="storage-initializer"
Jul 27 19:00:21.990187 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.990113    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="storage-initializer"
Jul 27 19:00:21.990536 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.990213    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8ac8353-1de3-4ba1-a2a3-f333427bd285" containerName="main"
Jul 27 19:00:21.990536 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.990228    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aeeb804-68d1-44c5-bd99-03fcae500228" containerName="main"
Jul 27 19:00:21.993296 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.993275    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:21.996001 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.995969    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Jul 27 19:00:21.996001 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.995973    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-6qx54\""
Jul 27 19:00:21.996199 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.996027    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Jul 27 19:00:21.996199 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.996046    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 19:00:21.996199 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.995976    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 27 19:00:21.996199 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:21.996000    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Jul 27 19:00:22.003264 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.003241    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"]
Jul 27 19:00:22.063349 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.063312    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-home\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.063349 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.063348    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-dshm\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.063575 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.063370    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-model-cache\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.063575 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.063449    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmxb2\" (UniqueName: \"kubernetes.io/projected/b0c4a929-3abb-4c3b-a865-0bfd270197e2-kube-api-access-zmxb2\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.063575 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.063524    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.063575 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.063549    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.063575 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.063572    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0c4a929-3abb-4c3b-a865-0bfd270197e2-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.063744 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.063627    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c4a929-3abb-4c3b-a865-0bfd270197e2-tls-certs\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.164371 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.164326    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.164371 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.164377    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0c4a929-3abb-4c3b-a865-0bfd270197e2-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.164642 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.164407    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c4a929-3abb-4c3b-a865-0bfd270197e2-tls-certs\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.164642 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.164448    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-home\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.164642 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.164463    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-dshm\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.164642 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.164519    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-model-cache\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.164642 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.164551    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmxb2\" (UniqueName: \"kubernetes.io/projected/b0c4a929-3abb-4c3b-a865-0bfd270197e2-kube-api-access-zmxb2\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.164642 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.164597    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.164957 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.164806    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.164957 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.164903    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-home\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.165078 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.164999    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-model-cache\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.165154 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.165135    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.165310 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.165290    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0c4a929-3abb-4c3b-a865-0bfd270197e2-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.166807 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.166787    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-dshm\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.167214 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.167192    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c4a929-3abb-4c3b-a865-0bfd270197e2-tls-certs\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.172660 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.172636    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmxb2\" (UniqueName: \"kubernetes.io/projected/b0c4a929-3abb-4c3b-a865-0bfd270197e2-kube-api-access-zmxb2\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.305108 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.305012    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:22.449414 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.449381    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"]
Jul 27 19:00:22.453115 ip-10-0-131-74 kubenswrapper[2571]: W0727 19:00:22.453083    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c4a929_3abb_4c3b_a865_0bfd270197e2.slice/crio-c11dea9c2939018db66f7576fd3bf0c0ab7ce7de1f5e70a9b6be361e1711125f WatchSource:0}: Error finding container c11dea9c2939018db66f7576fd3bf0c0ab7ce7de1f5e70a9b6be361e1711125f: Status 404 returned error can't find the container with id c11dea9c2939018db66f7576fd3bf0c0ab7ce7de1f5e70a9b6be361e1711125f
Jul 27 19:00:22.964333 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.964285    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" event={"ID":"b0c4a929-3abb-4c3b-a865-0bfd270197e2","Type":"ContainerStarted","Data":"3bdaa745725bda51495af8399efe68862b08b0200376acf6a4743fb7951add3d"}
Jul 27 19:00:22.964547 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:22.964394    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" event={"ID":"b0c4a929-3abb-4c3b-a865-0bfd270197e2","Type":"ContainerStarted","Data":"c11dea9c2939018db66f7576fd3bf0c0ab7ce7de1f5e70a9b6be361e1711125f"}
Jul 27 19:00:31.008404 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:31.008366    2571 generic.go:356] "Generic (PLEG): container finished" podID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerID="3bdaa745725bda51495af8399efe68862b08b0200376acf6a4743fb7951add3d" exitCode=0
Jul 27 19:00:31.008884 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:31.008420    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" event={"ID":"b0c4a929-3abb-4c3b-a865-0bfd270197e2","Type":"ContainerDied","Data":"3bdaa745725bda51495af8399efe68862b08b0200376acf6a4743fb7951add3d"}
Jul 27 19:00:32.016402 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:32.016365    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" event={"ID":"b0c4a929-3abb-4c3b-a865-0bfd270197e2","Type":"ContainerStarted","Data":"91287d0badd433dfe506bc2d5caa0ad66e061deee559369596a94635fde68c72"}
Jul 27 19:00:32.040212 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:32.040156    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" podStartSLOduration=3.159381774 podStartE2EDuration="11.040137583s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.880755809s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:00:32.036938166 +0000 UTC m=+5514.081676249" watchObservedRunningTime="2026-07-27 19:00:32.040137583 +0000 UTC m=+5514.084875665"
Jul 27 19:00:32.305275 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:32.305186    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:32.305275 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:32.305251    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:00:32.306379 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:32.306340    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 27 19:00:42.305867 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:42.305812    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 27 19:00:52.306242 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:00:52.306194    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 27 19:01:02.306418 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:01:02.306369    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 27 19:01:12.305796 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:01:12.305750    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 27 19:01:22.306729 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:01:22.306669    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 27 19:01:26.656575 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:01:26.656523    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 19:01:27.339033 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:01:27.338993    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 19:01:32.306060 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:01:32.305959    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 27 19:01:42.306132 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:01:42.306088    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 27 19:01:52.315714 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:01:52.315678    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:01:52.323638 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:01:52.323606    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:02:49.066594 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:02:49.066557    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7_b0c4a929-3abb-4c3b-a865-0bfd270197e2/storage-initializer/0.log"
Jul 27 19:02:49.078021 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:02:49.077961    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7_b0c4a929-3abb-4c3b-a865-0bfd270197e2/main/0.log"
Jul 27 19:02:53.020671 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:02:53.020627    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"]
Jul 27 19:03:08.043182 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.043120    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="main" containerID="cri-o://91287d0badd433dfe506bc2d5caa0ad66e061deee559369596a94635fde68c72" gracePeriod=45
Jul 27 19:03:08.317299 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.317198    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 27 19:03:08.701232 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.701201    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:03:08.829997 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.829962    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-tmp-dir\") pod \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") "
Jul 27 19:03:08.830209 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.830012    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-dshm\") pod \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") "
Jul 27 19:03:08.830209 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.830052    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-home\") pod \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") "
Jul 27 19:03:08.830209 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.830079    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-kserve-provision-location\") pod \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") "
Jul 27 19:03:08.830209 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.830111    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-model-cache\") pod \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") "
Jul 27 19:03:08.830209 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.830131    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmxb2\" (UniqueName: \"kubernetes.io/projected/b0c4a929-3abb-4c3b-a865-0bfd270197e2-kube-api-access-zmxb2\") pod \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") "
Jul 27 19:03:08.830209 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.830154    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c4a929-3abb-4c3b-a865-0bfd270197e2-tls-certs\") pod \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") "
Jul 27 19:03:08.830209 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.830199    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0c4a929-3abb-4c3b-a865-0bfd270197e2-cabundle-cert\") pod \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\" (UID: \"b0c4a929-3abb-4c3b-a865-0bfd270197e2\") "
Jul 27 19:03:08.830607 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.830457    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-home" (OuterVolumeSpecName: "home") pod "b0c4a929-3abb-4c3b-a865-0bfd270197e2" (UID: "b0c4a929-3abb-4c3b-a865-0bfd270197e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:03:08.830607 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.830474    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0c4a929-3abb-4c3b-a865-0bfd270197e2" (UID: "b0c4a929-3abb-4c3b-a865-0bfd270197e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:03:08.830848 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.830824    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c4a929-3abb-4c3b-a865-0bfd270197e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b0c4a929-3abb-4c3b-a865-0bfd270197e2" (UID: "b0c4a929-3abb-4c3b-a865-0bfd270197e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:03:08.832828 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.832799    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c4a929-3abb-4c3b-a865-0bfd270197e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0c4a929-3abb-4c3b-a865-0bfd270197e2" (UID: "b0c4a929-3abb-4c3b-a865-0bfd270197e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:03:08.832978 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.832946    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c4a929-3abb-4c3b-a865-0bfd270197e2-kube-api-access-zmxb2" (OuterVolumeSpecName: "kube-api-access-zmxb2") pod "b0c4a929-3abb-4c3b-a865-0bfd270197e2" (UID: "b0c4a929-3abb-4c3b-a865-0bfd270197e2"). InnerVolumeSpecName "kube-api-access-zmxb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:03:08.833171 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.833154    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-dshm" (OuterVolumeSpecName: "dshm") pod "b0c4a929-3abb-4c3b-a865-0bfd270197e2" (UID: "b0c4a929-3abb-4c3b-a865-0bfd270197e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:03:08.843160 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.843136    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0c4a929-3abb-4c3b-a865-0bfd270197e2" (UID: "b0c4a929-3abb-4c3b-a865-0bfd270197e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:03:08.890081 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.889993    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0c4a929-3abb-4c3b-a865-0bfd270197e2" (UID: "b0c4a929-3abb-4c3b-a865-0bfd270197e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:03:08.922701 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.922666    2571 generic.go:356] "Generic (PLEG): container finished" podID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerID="91287d0badd433dfe506bc2d5caa0ad66e061deee559369596a94635fde68c72" exitCode=0
Jul 27 19:03:08.922870 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.922734    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"
Jul 27 19:03:08.922870 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.922752    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" event={"ID":"b0c4a929-3abb-4c3b-a865-0bfd270197e2","Type":"ContainerDied","Data":"91287d0badd433dfe506bc2d5caa0ad66e061deee559369596a94635fde68c72"}
Jul 27 19:03:08.922870 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.922815    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7" event={"ID":"b0c4a929-3abb-4c3b-a865-0bfd270197e2","Type":"ContainerDied","Data":"c11dea9c2939018db66f7576fd3bf0c0ab7ce7de1f5e70a9b6be361e1711125f"}
Jul 27 19:03:08.922870 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.922848    2571 scope.go:117] "RemoveContainer" containerID="91287d0badd433dfe506bc2d5caa0ad66e061deee559369596a94635fde68c72"
Jul 27 19:03:08.930906 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.930884    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:03:08.930995 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.930913    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmxb2\" (UniqueName: \"kubernetes.io/projected/b0c4a929-3abb-4c3b-a865-0bfd270197e2-kube-api-access-zmxb2\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:03:08.930995 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.930927    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c4a929-3abb-4c3b-a865-0bfd270197e2-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:03:08.930995 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.930942    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0c4a929-3abb-4c3b-a865-0bfd270197e2-cabundle-cert\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:03:08.930995 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.930954    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:03:08.930995 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.930965    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:03:08.930995 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.930975    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:03:08.930995 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.930987    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c4a929-3abb-4c3b-a865-0bfd270197e2-kserve-provision-location\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:03:08.932315 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.932296    2571 scope.go:117] "RemoveContainer" containerID="3bdaa745725bda51495af8399efe68862b08b0200376acf6a4743fb7951add3d"
Jul 27 19:03:08.940569 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.940550    2571 scope.go:117] "RemoveContainer" containerID="91287d0badd433dfe506bc2d5caa0ad66e061deee559369596a94635fde68c72"
Jul 27 19:03:08.940798 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:03:08.940781    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91287d0badd433dfe506bc2d5caa0ad66e061deee559369596a94635fde68c72\": container with ID starting with 91287d0badd433dfe506bc2d5caa0ad66e061deee559369596a94635fde68c72 not found: ID does not exist" containerID="91287d0badd433dfe506bc2d5caa0ad66e061deee559369596a94635fde68c72"
Jul 27 19:03:08.940880 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.940810    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91287d0badd433dfe506bc2d5caa0ad66e061deee559369596a94635fde68c72"} err="failed to get container status \"91287d0badd433dfe506bc2d5caa0ad66e061deee559369596a94635fde68c72\": rpc error: code = NotFound desc = could not find container \"91287d0badd433dfe506bc2d5caa0ad66e061deee559369596a94635fde68c72\": container with ID starting with 91287d0badd433dfe506bc2d5caa0ad66e061deee559369596a94635fde68c72 not found: ID does not exist"
Jul 27 19:03:08.940880 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.940831    2571 scope.go:117] "RemoveContainer" containerID="3bdaa745725bda51495af8399efe68862b08b0200376acf6a4743fb7951add3d"
Jul 27 19:03:08.941044 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:03:08.941024    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bdaa745725bda51495af8399efe68862b08b0200376acf6a4743fb7951add3d\": container with ID starting with 3bdaa745725bda51495af8399efe68862b08b0200376acf6a4743fb7951add3d not found: ID does not exist" containerID="3bdaa745725bda51495af8399efe68862b08b0200376acf6a4743fb7951add3d"
Jul 27 19:03:08.941084 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.941052    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdaa745725bda51495af8399efe68862b08b0200376acf6a4743fb7951add3d"} err="failed to get container status \"3bdaa745725bda51495af8399efe68862b08b0200376acf6a4743fb7951add3d\": rpc error: code = NotFound desc = could not find container \"3bdaa745725bda51495af8399efe68862b08b0200376acf6a4743fb7951add3d\": container with ID starting with 3bdaa745725bda51495af8399efe68862b08b0200376acf6a4743fb7951add3d not found: ID does not exist"
Jul 27 19:03:08.944798 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.944777    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"]
Jul 27 19:03:08.948685 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:08.948664    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-pdsp7"]
Jul 27 19:03:10.399275 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:10.399241    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" path="/var/lib/kubelet/pods/b0c4a929-3abb-4c3b-a865-0bfd270197e2/volumes"
Jul 27 19:03:30.340101 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:03:30.340053    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 19:03:31.026892 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:31.026853    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 19:03:50.689053 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.689013    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"]
Jul 27 19:03:50.689574 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.689331    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="main"
Jul 27 19:03:50.689574 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.689342    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="main"
Jul 27 19:03:50.689574 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.689361    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="storage-initializer"
Jul 27 19:03:50.689574 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.689367    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="storage-initializer"
Jul 27 19:03:50.689574 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.689416    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0c4a929-3abb-4c3b-a865-0bfd270197e2" containerName="main"
Jul 27 19:03:50.693614 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.693590    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.696382 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.696358    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 27 19:03:50.696543 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.696452    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 27 19:03:50.696543 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.696532    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Jul 27 19:03:50.696687 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.696590    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 27 19:03:50.698070 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.698053    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-vgrrn\""
Jul 27 19:03:50.704129 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.704106    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"]
Jul 27 19:03:50.800430 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.800392    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-kserve-provision-location\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.800430 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.800435    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f4xp\" (UniqueName: \"kubernetes.io/projected/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-kube-api-access-5f4xp\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.800679 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.800473    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-dshm\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.800679 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.800547    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-model-cache\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.800679 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.800566    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-tmp-dir\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.800679 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.800598    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-tls-certs\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.800679 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.800615    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-home\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.893619 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.893584    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"]
Jul 27 19:03:50.896860 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.896841    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"
Jul 27 19:03:50.899365 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.899336    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-xml74\""
Jul 27 19:03:50.901051 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.901032    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-home\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.901126 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.901084    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-kserve-provision-location\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.901126 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.901112    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f4xp\" (UniqueName: \"kubernetes.io/projected/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-kube-api-access-5f4xp\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.901228 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.901150    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-dshm\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.901228 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.901208    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-model-cache\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.901335 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.901237    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-tmp-dir\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.901335 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.901286    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-tls-certs\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.901563 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.901541    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-home\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.901691 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.901670    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-model-cache\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.901784 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.901752    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-tmp-dir\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.901950 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.901917    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-kserve-provision-location\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.904835 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.904791    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-tls-certs\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.905008 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.904985    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-dshm\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:50.906408 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.906386    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"]
Jul 27 19:03:50.915917 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:50.915889    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f4xp\" (UniqueName: \"kubernetes.io/projected/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-kube-api-access-5f4xp\") pod \"tls-verification-test-kserve-b4c7b9b88-t9hvh\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:51.001972 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:51.001932    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e502c032-ed11-4b0b-bafd-93694ee16fe4-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-vb99x\" (UID: \"e502c032-ed11-4b0b-bafd-93694ee16fe4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"
Jul 27 19:03:51.002161 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:51.002098    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dz72\" (UniqueName: \"kubernetes.io/projected/e502c032-ed11-4b0b-bafd-93694ee16fe4-kube-api-access-6dz72\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-vb99x\" (UID: \"e502c032-ed11-4b0b-bafd-93694ee16fe4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"
Jul 27 19:03:51.004857 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:51.004834    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:03:51.103551 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:51.103514    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dz72\" (UniqueName: \"kubernetes.io/projected/e502c032-ed11-4b0b-bafd-93694ee16fe4-kube-api-access-6dz72\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-vb99x\" (UID: \"e502c032-ed11-4b0b-bafd-93694ee16fe4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"
Jul 27 19:03:51.103816 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:51.103601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e502c032-ed11-4b0b-bafd-93694ee16fe4-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-vb99x\" (UID: \"e502c032-ed11-4b0b-bafd-93694ee16fe4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"
Jul 27 19:03:51.106190 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:51.106159    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e502c032-ed11-4b0b-bafd-93694ee16fe4-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-vb99x\" (UID: \"e502c032-ed11-4b0b-bafd-93694ee16fe4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"
Jul 27 19:03:51.114436 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:51.114406    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dz72\" (UniqueName: \"kubernetes.io/projected/e502c032-ed11-4b0b-bafd-93694ee16fe4-kube-api-access-6dz72\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-vb99x\" (UID: \"e502c032-ed11-4b0b-bafd-93694ee16fe4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"
Jul 27 19:03:51.161560 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:51.161252    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"]
Jul 27 19:03:51.165112 ip-10-0-131-74 kubenswrapper[2571]: W0727 19:03:51.165082    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5a10a0_cdf6_4493_ae5f_313c6b46aabd.slice/crio-deb24f8e983bcd983cb9facdf16e4c1aa47e2051160760165bb1a6937ca51e67 WatchSource:0}: Error finding container deb24f8e983bcd983cb9facdf16e4c1aa47e2051160760165bb1a6937ca51e67: Status 404 returned error can't find the container with id deb24f8e983bcd983cb9facdf16e4c1aa47e2051160760165bb1a6937ca51e67
Jul 27 19:03:51.226619 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:51.226597    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"
Jul 27 19:03:51.359360 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:51.359334    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"]
Jul 27 19:03:51.360807 ip-10-0-131-74 kubenswrapper[2571]: W0727 19:03:51.360777    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode502c032_ed11_4b0b_bafd_93694ee16fe4.slice/crio-7cc19ac6b8d9851f1f1081f986ca54437729710fe075ddb954baddfd384ca990 WatchSource:0}: Error finding container 7cc19ac6b8d9851f1f1081f986ca54437729710fe075ddb954baddfd384ca990: Status 404 returned error can't find the container with id 7cc19ac6b8d9851f1f1081f986ca54437729710fe075ddb954baddfd384ca990
Jul 27 19:03:52.142947 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:52.142809    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x" event={"ID":"e502c032-ed11-4b0b-bafd-93694ee16fe4","Type":"ContainerStarted","Data":"2caaf6e139096d9c4a6b574a2233b7ad65219ffdf1dfef7ddd8eb1e5c34d59e6"}
Jul 27 19:03:52.142947 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:52.142908    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x" event={"ID":"e502c032-ed11-4b0b-bafd-93694ee16fe4","Type":"ContainerStarted","Data":"7cc19ac6b8d9851f1f1081f986ca54437729710fe075ddb954baddfd384ca990"}
Jul 27 19:03:52.143608 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:52.142970    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"
Jul 27 19:03:52.145029 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:52.144998    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" event={"ID":"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd","Type":"ContainerStarted","Data":"2a830501c6bbb5499fd11e01981549f939eaae4307299432c1d7a1472cb872a4"}
Jul 27 19:03:52.145168 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:52.145051    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" event={"ID":"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd","Type":"ContainerStarted","Data":"deb24f8e983bcd983cb9facdf16e4c1aa47e2051160760165bb1a6937ca51e67"}
Jul 27 19:03:52.162352 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:52.162289    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x" podStartSLOduration=2.162272173 podStartE2EDuration="2.162272173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:03:52.15939706 +0000 UTC m=+5714.204135142" watchObservedRunningTime="2026-07-27 19:03:52.162272173 +0000 UTC m=+5714.207010253"
Jul 27 19:03:58.179226 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:58.179187    2571 generic.go:356] "Generic (PLEG): container finished" podID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerID="2a830501c6bbb5499fd11e01981549f939eaae4307299432c1d7a1472cb872a4" exitCode=0
Jul 27 19:03:58.179690 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:58.179261    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" event={"ID":"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd","Type":"ContainerDied","Data":"2a830501c6bbb5499fd11e01981549f939eaae4307299432c1d7a1472cb872a4"}
Jul 27 19:03:59.189653 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:59.189614    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" event={"ID":"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd","Type":"ContainerStarted","Data":"8bae405c6aa14b4bfb4c09a48a937bdce262a31da144b9f5755fe8e152a6c355"}
Jul 27 19:03:59.213030 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:03:59.212967    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" podStartSLOduration=2.9440449749999997 podStartE2EDuration="9.212950493s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.268905518s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:03:59.209759308 +0000 UTC m=+5721.254497401" watchObservedRunningTime="2026-07-27 19:03:59.212950493 +0000 UTC m=+5721.257688574"
Jul 27 19:04:00.004928 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:04:00.004892    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:04:01.005350 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:04:01.005308    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:04:01.006764 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:04:01.006720    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 27 19:04:11.006025 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:04:11.005972    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 27 19:04:21.006063 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:04:21.006016    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 27 19:04:23.152238 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:04:23.152206    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"
Jul 27 19:04:31.005892 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:04:31.005786    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 27 19:04:41.005729 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:04:41.005686    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 27 19:04:51.006152 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:04:51.006104    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 27 19:05:01.005991 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:05:01.005934    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 27 19:05:11.005793 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:05:11.005747    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 27 19:05:21.005778 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:05:21.005727    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 27 19:05:31.005691 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:05:31.005642    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 27 19:05:34.027865 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:05:34.027818    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 19:05:34.704063 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:05:34.704019    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 19:05:41.005224 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:05:41.005179    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 27 19:05:51.015238 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:05:51.015204    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:05:51.023147 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:05:51.023119    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:06:16.097019 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:16.096985    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"]
Jul 27 19:06:16.104722 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:16.104681    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"]
Jul 27 19:06:23.152527 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:23.152465    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x" podUID="e502c032-ed11-4b0b-bafd-93694ee16fe4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 19:06:31.119013 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.118963    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x" podUID="e502c032-ed11-4b0b-bafd-93694ee16fe4" containerName="main" containerID="cri-o://2caaf6e139096d9c4a6b574a2233b7ad65219ffdf1dfef7ddd8eb1e5c34d59e6" gracePeriod=45
Jul 27 19:06:31.127559 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.127515    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main" containerID="cri-o://8bae405c6aa14b4bfb4c09a48a937bdce262a31da144b9f5755fe8e152a6c355" gracePeriod=45
Jul 27 19:06:31.378878 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.378804    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"
Jul 27 19:06:31.451652 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.451613    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e502c032-ed11-4b0b-bafd-93694ee16fe4-tls-certs\") pod \"e502c032-ed11-4b0b-bafd-93694ee16fe4\" (UID: \"e502c032-ed11-4b0b-bafd-93694ee16fe4\") "
Jul 27 19:06:31.451652 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.451662    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dz72\" (UniqueName: \"kubernetes.io/projected/e502c032-ed11-4b0b-bafd-93694ee16fe4-kube-api-access-6dz72\") pod \"e502c032-ed11-4b0b-bafd-93694ee16fe4\" (UID: \"e502c032-ed11-4b0b-bafd-93694ee16fe4\") "
Jul 27 19:06:31.454009 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.453970    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e502c032-ed11-4b0b-bafd-93694ee16fe4-kube-api-access-6dz72" (OuterVolumeSpecName: "kube-api-access-6dz72") pod "e502c032-ed11-4b0b-bafd-93694ee16fe4" (UID: "e502c032-ed11-4b0b-bafd-93694ee16fe4"). InnerVolumeSpecName "kube-api-access-6dz72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:06:31.454153 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.454090    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e502c032-ed11-4b0b-bafd-93694ee16fe4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e502c032-ed11-4b0b-bafd-93694ee16fe4" (UID: "e502c032-ed11-4b0b-bafd-93694ee16fe4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:06:31.553207 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.553166    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e502c032-ed11-4b0b-bafd-93694ee16fe4-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:06:31.553207 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.553199    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dz72\" (UniqueName: \"kubernetes.io/projected/e502c032-ed11-4b0b-bafd-93694ee16fe4-kube-api-access-6dz72\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:06:31.862436 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.862411    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:06:31.956971 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.956942    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-tmp-dir\") pod \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") "
Jul 27 19:06:31.957173 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.956986    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-dshm\") pod \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") "
Jul 27 19:06:31.957173 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.957004    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-tls-certs\") pod \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") "
Jul 27 19:06:31.957173 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.957022    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f4xp\" (UniqueName: \"kubernetes.io/projected/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-kube-api-access-5f4xp\") pod \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") "
Jul 27 19:06:31.957361 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.957234    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-kserve-provision-location\") pod \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") "
Jul 27 19:06:31.957361 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.957292    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-home\") pod \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") "
Jul 27 19:06:31.957361 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.957337    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-model-cache\") pod \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\" (UID: \"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd\") "
Jul 27 19:06:31.957866 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.957835    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" (UID: "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:06:31.958140 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.958111    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-home" (OuterVolumeSpecName: "home") pod "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" (UID: "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:06:31.959340 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.959307    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" (UID: "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:06:31.959340 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.959327    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-dshm" (OuterVolumeSpecName: "dshm") pod "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" (UID: "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:06:31.959609 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.959588    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-kube-api-access-5f4xp" (OuterVolumeSpecName: "kube-api-access-5f4xp") pod "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" (UID: "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd"). InnerVolumeSpecName "kube-api-access-5f4xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:06:31.969714 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.969689    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" (UID: "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:06:31.991439 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.991412    2571 generic.go:356] "Generic (PLEG): container finished" podID="e502c032-ed11-4b0b-bafd-93694ee16fe4" containerID="2caaf6e139096d9c4a6b574a2233b7ad65219ffdf1dfef7ddd8eb1e5c34d59e6" exitCode=0
Jul 27 19:06:31.991602 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.991466    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"
Jul 27 19:06:31.991602 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.991508    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x" event={"ID":"e502c032-ed11-4b0b-bafd-93694ee16fe4","Type":"ContainerDied","Data":"2caaf6e139096d9c4a6b574a2233b7ad65219ffdf1dfef7ddd8eb1e5c34d59e6"}
Jul 27 19:06:31.991725 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.991602    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x" event={"ID":"e502c032-ed11-4b0b-bafd-93694ee16fe4","Type":"ContainerDied","Data":"7cc19ac6b8d9851f1f1081f986ca54437729710fe075ddb954baddfd384ca990"}
Jul 27 19:06:31.991725 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.991631    2571 scope.go:117] "RemoveContainer" containerID="2caaf6e139096d9c4a6b574a2233b7ad65219ffdf1dfef7ddd8eb1e5c34d59e6"
Jul 27 19:06:31.993321 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.993289    2571 generic.go:356] "Generic (PLEG): container finished" podID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerID="8bae405c6aa14b4bfb4c09a48a937bdce262a31da144b9f5755fe8e152a6c355" exitCode=0
Jul 27 19:06:31.993420 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.993356    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"
Jul 27 19:06:31.993420 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.993352    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" event={"ID":"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd","Type":"ContainerDied","Data":"8bae405c6aa14b4bfb4c09a48a937bdce262a31da144b9f5755fe8e152a6c355"}
Jul 27 19:06:31.993542 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:31.993418    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh" event={"ID":"0a5a10a0-cdf6-4493-ae5f-313c6b46aabd","Type":"ContainerDied","Data":"deb24f8e983bcd983cb9facdf16e4c1aa47e2051160760165bb1a6937ca51e67"}
Jul 27 19:06:32.000954 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.000933    2571 scope.go:117] "RemoveContainer" containerID="2caaf6e139096d9c4a6b574a2233b7ad65219ffdf1dfef7ddd8eb1e5c34d59e6"
Jul 27 19:06:32.001281 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:06:32.001252    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2caaf6e139096d9c4a6b574a2233b7ad65219ffdf1dfef7ddd8eb1e5c34d59e6\": container with ID starting with 2caaf6e139096d9c4a6b574a2233b7ad65219ffdf1dfef7ddd8eb1e5c34d59e6 not found: ID does not exist" containerID="2caaf6e139096d9c4a6b574a2233b7ad65219ffdf1dfef7ddd8eb1e5c34d59e6"
Jul 27 19:06:32.001342 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.001293    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2caaf6e139096d9c4a6b574a2233b7ad65219ffdf1dfef7ddd8eb1e5c34d59e6"} err="failed to get container status \"2caaf6e139096d9c4a6b574a2233b7ad65219ffdf1dfef7ddd8eb1e5c34d59e6\": rpc error: code = NotFound desc = could not find container \"2caaf6e139096d9c4a6b574a2233b7ad65219ffdf1dfef7ddd8eb1e5c34d59e6\": container with ID starting with 2caaf6e139096d9c4a6b574a2233b7ad65219ffdf1dfef7ddd8eb1e5c34d59e6 not found: ID does not exist"
Jul 27 19:06:32.001342 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.001316    2571 scope.go:117] "RemoveContainer" containerID="8bae405c6aa14b4bfb4c09a48a937bdce262a31da144b9f5755fe8e152a6c355"
Jul 27 19:06:32.010932 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.010913    2571 scope.go:117] "RemoveContainer" containerID="2a830501c6bbb5499fd11e01981549f939eaae4307299432c1d7a1472cb872a4"
Jul 27 19:06:32.014408 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.014381    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"]
Jul 27 19:06:32.016609 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.016582    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-vb99x"]
Jul 27 19:06:32.019728 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.019713    2571 scope.go:117] "RemoveContainer" containerID="8bae405c6aa14b4bfb4c09a48a937bdce262a31da144b9f5755fe8e152a6c355"
Jul 27 19:06:32.020042 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:06:32.020023    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bae405c6aa14b4bfb4c09a48a937bdce262a31da144b9f5755fe8e152a6c355\": container with ID starting with 8bae405c6aa14b4bfb4c09a48a937bdce262a31da144b9f5755fe8e152a6c355 not found: ID does not exist" containerID="8bae405c6aa14b4bfb4c09a48a937bdce262a31da144b9f5755fe8e152a6c355"
Jul 27 19:06:32.020101 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.020049    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bae405c6aa14b4bfb4c09a48a937bdce262a31da144b9f5755fe8e152a6c355"} err="failed to get container status \"8bae405c6aa14b4bfb4c09a48a937bdce262a31da144b9f5755fe8e152a6c355\": rpc error: code = NotFound desc = could not find container \"8bae405c6aa14b4bfb4c09a48a937bdce262a31da144b9f5755fe8e152a6c355\": container with ID starting with 8bae405c6aa14b4bfb4c09a48a937bdce262a31da144b9f5755fe8e152a6c355 not found: ID does not exist"
Jul 27 19:06:32.020101 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.020066    2571 scope.go:117] "RemoveContainer" containerID="2a830501c6bbb5499fd11e01981549f939eaae4307299432c1d7a1472cb872a4"
Jul 27 19:06:32.020303 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:06:32.020283    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a830501c6bbb5499fd11e01981549f939eaae4307299432c1d7a1472cb872a4\": container with ID starting with 2a830501c6bbb5499fd11e01981549f939eaae4307299432c1d7a1472cb872a4 not found: ID does not exist" containerID="2a830501c6bbb5499fd11e01981549f939eaae4307299432c1d7a1472cb872a4"
Jul 27 19:06:32.020361 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.020313    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a830501c6bbb5499fd11e01981549f939eaae4307299432c1d7a1472cb872a4"} err="failed to get container status \"2a830501c6bbb5499fd11e01981549f939eaae4307299432c1d7a1472cb872a4\": rpc error: code = NotFound desc = could not find container \"2a830501c6bbb5499fd11e01981549f939eaae4307299432c1d7a1472cb872a4\": container with ID starting with 2a830501c6bbb5499fd11e01981549f939eaae4307299432c1d7a1472cb872a4 not found: ID does not exist"
Jul 27 19:06:32.024729 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.024704    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" (UID: "0a5a10a0-cdf6-4493-ae5f-313c6b46aabd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:06:32.058877 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.058841    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-kserve-provision-location\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:06:32.058877 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.058874    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:06:32.059045 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.058887    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:06:32.059045 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.058896    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:06:32.059045 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.058905    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:06:32.059045 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.058914    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:06:32.059045 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.058923    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f4xp\" (UniqueName: \"kubernetes.io/projected/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd-kube-api-access-5f4xp\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:06:32.318169 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.318133    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"]
Jul 27 19:06:32.321565 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.321539    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-t9hvh"]
Jul 27 19:06:32.399519 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.399470    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" path="/var/lib/kubelet/pods/0a5a10a0-cdf6-4493-ae5f-313c6b46aabd/volumes"
Jul 27 19:06:32.400044 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:32.400021    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e502c032-ed11-4b0b-bafd-93694ee16fe4" path="/var/lib/kubelet/pods/e502c032-ed11-4b0b-bafd-93694ee16fe4/volumes"
Jul 27 19:06:44.713374 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.713336    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"]
Jul 27 19:06:44.713820 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.713655    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="storage-initializer"
Jul 27 19:06:44.713820 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.713667    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="storage-initializer"
Jul 27 19:06:44.713820 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.713681    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main"
Jul 27 19:06:44.713820 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.713687    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main"
Jul 27 19:06:44.713820 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.713705    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e502c032-ed11-4b0b-bafd-93694ee16fe4" containerName="main"
Jul 27 19:06:44.713820 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.713710    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e502c032-ed11-4b0b-bafd-93694ee16fe4" containerName="main"
Jul 27 19:06:44.713820 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.713757    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e502c032-ed11-4b0b-bafd-93694ee16fe4" containerName="main"
Jul 27 19:06:44.713820 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.713765    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a5a10a0-cdf6-4493-ae5f-313c6b46aabd" containerName="main"
Jul 27 19:06:44.716743 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.716726    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.719400 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.719366    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 27 19:06:44.719400 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.719387    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 27 19:06:44.719619 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.719413    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-6685r\""
Jul 27 19:06:44.719768 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.719751    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 27 19:06:44.719849 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.719763    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 27 19:06:44.727362 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.727342    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"]
Jul 27 19:06:44.772573 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.772542    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-model-cache\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.772723 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.772597    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-kserve-provision-location\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.772723 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.772629    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-dshm\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.772723 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.772679    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8731e06c-285f-48c3-bffd-bf0ead6dd899-tls-certs\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.772827 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.772723    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-tmp-dir\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.772827 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.772796    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqzjd\" (UniqueName: \"kubernetes.io/projected/8731e06c-285f-48c3-bffd-bf0ead6dd899-kube-api-access-nqzjd\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.772893 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.772828    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-home\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.873616 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.873571    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-tmp-dir\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.873824 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.873644    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqzjd\" (UniqueName: \"kubernetes.io/projected/8731e06c-285f-48c3-bffd-bf0ead6dd899-kube-api-access-nqzjd\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.873824 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.873662    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-home\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.873824 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.873691    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-model-cache\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.873824 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.873729    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-kserve-provision-location\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.874070 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.873853    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-dshm\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.874070 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.873900    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8731e06c-285f-48c3-bffd-bf0ead6dd899-tls-certs\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.874070 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.874040    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-tmp-dir\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.874229 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.874062    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-model-cache\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.874229 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.874124    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-kserve-provision-location\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.874229 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.874158    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-home\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.876135 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.876116    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-dshm\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.876409 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.876393    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8731e06c-285f-48c3-bffd-bf0ead6dd899-tls-certs\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:44.881324 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:44.881298    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqzjd\" (UniqueName: \"kubernetes.io/projected/8731e06c-285f-48c3-bffd-bf0ead6dd899-kube-api-access-nqzjd\") pod \"prestop-hook-test-kserve-54c766cd44-lfd99\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:45.028217 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:45.028125    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:06:45.165695 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:45.165671    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"]
Jul 27 19:06:45.167543 ip-10-0-131-74 kubenswrapper[2571]: W0727 19:06:45.167508    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8731e06c_285f_48c3_bffd_bf0ead6dd899.slice/crio-e10699af101b92da55cb80463161317089f9be19679290b81f7b20fd951d8cb2 WatchSource:0}: Error finding container e10699af101b92da55cb80463161317089f9be19679290b81f7b20fd951d8cb2: Status 404 returned error can't find the container with id e10699af101b92da55cb80463161317089f9be19679290b81f7b20fd951d8cb2
Jul 27 19:06:46.064214 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:46.064158    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" event={"ID":"8731e06c-285f-48c3-bffd-bf0ead6dd899","Type":"ContainerStarted","Data":"fb34129104c826b0273243e408aa79046ba814c31cfae03c4e5e6080e07de131"}
Jul 27 19:06:46.064687 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:46.064241    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" event={"ID":"8731e06c-285f-48c3-bffd-bf0ead6dd899","Type":"ContainerStarted","Data":"e10699af101b92da55cb80463161317089f9be19679290b81f7b20fd951d8cb2"}
Jul 27 19:06:54.103944 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:54.103903    2571 generic.go:356] "Generic (PLEG): container finished" podID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerID="fb34129104c826b0273243e408aa79046ba814c31cfae03c4e5e6080e07de131" exitCode=0
Jul 27 19:06:54.104443 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:54.103989    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" event={"ID":"8731e06c-285f-48c3-bffd-bf0ead6dd899","Type":"ContainerDied","Data":"fb34129104c826b0273243e408aa79046ba814c31cfae03c4e5e6080e07de131"}
Jul 27 19:06:55.111038 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:55.110991    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" event={"ID":"8731e06c-285f-48c3-bffd-bf0ead6dd899","Type":"ContainerStarted","Data":"d4a49b66b1a11d6cad144599affb59f1895e5c2997218ceef38956f2c643f5fb"}
Jul 27 19:06:55.138119 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:55.138056    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" podStartSLOduration=2.950527516 podStartE2EDuration="11.138039028s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.187511512s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:06:55.136651799 +0000 UTC m=+5897.181389880" watchObservedRunningTime="2026-07-27 19:06:55.138039028 +0000 UTC m=+5897.182777157"
Jul 27 19:06:56.029022 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:06:56.028982    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:07:05.029096 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:07:05.029049    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:07:05.030674 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:07:05.030643    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 27 19:07:15.028825 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:07:15.028782    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 27 19:07:25.028713 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:07:25.028649    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 27 19:07:35.029674 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:07:35.029566    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 27 19:07:37.704903 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:07:37.704845    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 19:07:38.337166 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:07:38.337132    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 19:07:45.029549 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:07:45.029485    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 27 19:07:55.029572 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:07:55.029523    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 27 19:08:05.029223 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:08:05.029177    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 27 19:08:15.029428 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:08:15.029381    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 27 19:08:25.029172 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:08:25.029126    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 27 19:08:35.028908 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:08:35.028862    2571 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 27 19:08:45.039088 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:08:45.039057    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:08:45.046931 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:08:45.046905    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:08:46.472944 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:08:46.472912    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"]
Jul 27 19:09:01.710961 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:01.710851    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main" containerID="cri-o://d4a49b66b1a11d6cad144599affb59f1895e5c2997218ceef38956f2c643f5fb" gracePeriod=45
Jul 27 19:09:02.039793 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.039694    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 27 19:09:02.390681 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.390656    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:09:02.402212 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.402184    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-dshm\") pod \"8731e06c-285f-48c3-bffd-bf0ead6dd899\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") "
Jul 27 19:09:02.402345 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.402225    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8731e06c-285f-48c3-bffd-bf0ead6dd899-tls-certs\") pod \"8731e06c-285f-48c3-bffd-bf0ead6dd899\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") "
Jul 27 19:09:02.402345 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.402251    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-tmp-dir\") pod \"8731e06c-285f-48c3-bffd-bf0ead6dd899\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") "
Jul 27 19:09:02.402345 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.402278    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-kserve-provision-location\") pod \"8731e06c-285f-48c3-bffd-bf0ead6dd899\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") "
Jul 27 19:09:02.402345 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.402320    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-home\") pod \"8731e06c-285f-48c3-bffd-bf0ead6dd899\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") "
Jul 27 19:09:02.402345 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.402342    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqzjd\" (UniqueName: \"kubernetes.io/projected/8731e06c-285f-48c3-bffd-bf0ead6dd899-kube-api-access-nqzjd\") pod \"8731e06c-285f-48c3-bffd-bf0ead6dd899\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") "
Jul 27 19:09:02.402668 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.402422    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-model-cache\") pod \"8731e06c-285f-48c3-bffd-bf0ead6dd899\" (UID: \"8731e06c-285f-48c3-bffd-bf0ead6dd899\") "
Jul 27 19:09:02.403054 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.402869    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-model-cache" (OuterVolumeSpecName: "model-cache") pod "8731e06c-285f-48c3-bffd-bf0ead6dd899" (UID: "8731e06c-285f-48c3-bffd-bf0ead6dd899"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:09:02.403274 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.403244    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-home" (OuterVolumeSpecName: "home") pod "8731e06c-285f-48c3-bffd-bf0ead6dd899" (UID: "8731e06c-285f-48c3-bffd-bf0ead6dd899"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:09:02.405195 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.405171    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-dshm" (OuterVolumeSpecName: "dshm") pod "8731e06c-285f-48c3-bffd-bf0ead6dd899" (UID: "8731e06c-285f-48c3-bffd-bf0ead6dd899"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:09:02.405451 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.405392    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8731e06c-285f-48c3-bffd-bf0ead6dd899-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8731e06c-285f-48c3-bffd-bf0ead6dd899" (UID: "8731e06c-285f-48c3-bffd-bf0ead6dd899"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:09:02.405570 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.405553    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8731e06c-285f-48c3-bffd-bf0ead6dd899-kube-api-access-nqzjd" (OuterVolumeSpecName: "kube-api-access-nqzjd") pod "8731e06c-285f-48c3-bffd-bf0ead6dd899" (UID: "8731e06c-285f-48c3-bffd-bf0ead6dd899"). InnerVolumeSpecName "kube-api-access-nqzjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:09:02.415136 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.415110    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8731e06c-285f-48c3-bffd-bf0ead6dd899" (UID: "8731e06c-285f-48c3-bffd-bf0ead6dd899"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:09:02.471555 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.471479    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8731e06c-285f-48c3-bffd-bf0ead6dd899" (UID: "8731e06c-285f-48c3-bffd-bf0ead6dd899"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:09:02.503577 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.503543    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqzjd\" (UniqueName: \"kubernetes.io/projected/8731e06c-285f-48c3-bffd-bf0ead6dd899-kube-api-access-nqzjd\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:09:02.503577 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.503567    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:09:02.503577 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.503577    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:09:02.503577 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.503585    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8731e06c-285f-48c3-bffd-bf0ead6dd899-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:09:02.503893 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.503593    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:09:02.503893 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.503601    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-kserve-provision-location\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:09:02.503893 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.503608    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8731e06c-285f-48c3-bffd-bf0ead6dd899-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:09:02.768345 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.768308    2571 generic.go:356] "Generic (PLEG): container finished" podID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerID="d4a49b66b1a11d6cad144599affb59f1895e5c2997218ceef38956f2c643f5fb" exitCode=0
Jul 27 19:09:02.768817 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.768361    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"
Jul 27 19:09:02.768817 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.768388    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" event={"ID":"8731e06c-285f-48c3-bffd-bf0ead6dd899","Type":"ContainerDied","Data":"d4a49b66b1a11d6cad144599affb59f1895e5c2997218ceef38956f2c643f5fb"}
Jul 27 19:09:02.768817 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.768440    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99" event={"ID":"8731e06c-285f-48c3-bffd-bf0ead6dd899","Type":"ContainerDied","Data":"e10699af101b92da55cb80463161317089f9be19679290b81f7b20fd951d8cb2"}
Jul 27 19:09:02.768817 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.768463    2571 scope.go:117] "RemoveContainer" containerID="d4a49b66b1a11d6cad144599affb59f1895e5c2997218ceef38956f2c643f5fb"
Jul 27 19:09:02.782614 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.782595    2571 scope.go:117] "RemoveContainer" containerID="fb34129104c826b0273243e408aa79046ba814c31cfae03c4e5e6080e07de131"
Jul 27 19:09:02.790948 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.790930    2571 scope.go:117] "RemoveContainer" containerID="d4a49b66b1a11d6cad144599affb59f1895e5c2997218ceef38956f2c643f5fb"
Jul 27 19:09:02.791170 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:09:02.791152    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a49b66b1a11d6cad144599affb59f1895e5c2997218ceef38956f2c643f5fb\": container with ID starting with d4a49b66b1a11d6cad144599affb59f1895e5c2997218ceef38956f2c643f5fb not found: ID does not exist" containerID="d4a49b66b1a11d6cad144599affb59f1895e5c2997218ceef38956f2c643f5fb"
Jul 27 19:09:02.791228 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.791176    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a49b66b1a11d6cad144599affb59f1895e5c2997218ceef38956f2c643f5fb"} err="failed to get container status \"d4a49b66b1a11d6cad144599affb59f1895e5c2997218ceef38956f2c643f5fb\": rpc error: code = NotFound desc = could not find container \"d4a49b66b1a11d6cad144599affb59f1895e5c2997218ceef38956f2c643f5fb\": container with ID starting with d4a49b66b1a11d6cad144599affb59f1895e5c2997218ceef38956f2c643f5fb not found: ID does not exist"
Jul 27 19:09:02.791228 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.791193    2571 scope.go:117] "RemoveContainer" containerID="fb34129104c826b0273243e408aa79046ba814c31cfae03c4e5e6080e07de131"
Jul 27 19:09:02.791403 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:09:02.791387    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb34129104c826b0273243e408aa79046ba814c31cfae03c4e5e6080e07de131\": container with ID starting with fb34129104c826b0273243e408aa79046ba814c31cfae03c4e5e6080e07de131 not found: ID does not exist" containerID="fb34129104c826b0273243e408aa79046ba814c31cfae03c4e5e6080e07de131"
Jul 27 19:09:02.791446 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.791406    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb34129104c826b0273243e408aa79046ba814c31cfae03c4e5e6080e07de131"} err="failed to get container status \"fb34129104c826b0273243e408aa79046ba814c31cfae03c4e5e6080e07de131\": rpc error: code = NotFound desc = could not find container \"fb34129104c826b0273243e408aa79046ba814c31cfae03c4e5e6080e07de131\": container with ID starting with fb34129104c826b0273243e408aa79046ba814c31cfae03c4e5e6080e07de131 not found: ID does not exist"
Jul 27 19:09:02.794651 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.794629    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"]
Jul 27 19:09:02.798948 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:02.798922    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-lfd99"]
Jul 27 19:09:04.398998 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:04.398968    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" path="/var/lib/kubelet/pods/8731e06c-285f-48c3-bffd-bf0ead6dd899/volumes"
Jul 27 19:09:41.337984 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:09:41.337941    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 19:09:41.961640 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:09:41.961601    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 19:10:26.719145 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.719111    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"]
Jul 27 19:10:26.719675 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.719413    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="storage-initializer"
Jul 27 19:10:26.719675 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.719424    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="storage-initializer"
Jul 27 19:10:26.719675 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.719441    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main"
Jul 27 19:10:26.719675 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.719447    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main"
Jul 27 19:10:26.719675 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.719509    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8731e06c-285f-48c3-bffd-bf0ead6dd899" containerName="main"
Jul 27 19:10:26.722311 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.722293    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.726912 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.726891    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 27 19:10:26.728134 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.728116    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 27 19:10:26.728232 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.728201    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-4mb28\""
Jul 27 19:10:26.728309 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.728292    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 27 19:10:26.733034 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.733011    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"]
Jul 27 19:10:26.791203 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.791168    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fszt\" (UniqueName: \"kubernetes.io/projected/c6a041af-56a8-423a-aa4a-d70f88eb3d85-kube-api-access-9fszt\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.791203 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.791205    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.791386 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.791263    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.791386 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.791309    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a041af-56a8-423a-aa4a-d70f88eb3d85-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.791386 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.791332    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.791386 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.791355    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.892456 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.892407    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fszt\" (UniqueName: \"kubernetes.io/projected/c6a041af-56a8-423a-aa4a-d70f88eb3d85-kube-api-access-9fszt\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.892456 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.892459    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.892715 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.892545    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.892715 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.892589    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a041af-56a8-423a-aa4a-d70f88eb3d85-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.892715 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.892616    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.892715 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.892643    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.892951 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.892861    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.893033 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.893005    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.893100 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.893080    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.894961 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.894935    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.895257 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.895232    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a041af-56a8-423a-aa4a-d70f88eb3d85-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:26.901357 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:26.901338    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fszt\" (UniqueName: \"kubernetes.io/projected/c6a041af-56a8-423a-aa4a-d70f88eb3d85-kube-api-access-9fszt\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qf7xg\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:27.033324 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:27.033229    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:27.181782 ip-10-0-131-74 kubenswrapper[2571]: W0727 19:10:27.181752    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a041af_56a8_423a_aa4a_d70f88eb3d85.slice/crio-368a94abe8a26c8b8ce8fab55944d627ff6709311275485d88d6ca64e1d6becf WatchSource:0}: Error finding container 368a94abe8a26c8b8ce8fab55944d627ff6709311275485d88d6ca64e1d6becf: Status 404 returned error can't find the container with id 368a94abe8a26c8b8ce8fab55944d627ff6709311275485d88d6ca64e1d6becf
Jul 27 19:10:27.184993 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:27.184968    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"]
Jul 27 19:10:28.180657 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:28.180620    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg" event={"ID":"c6a041af-56a8-423a-aa4a-d70f88eb3d85","Type":"ContainerStarted","Data":"bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b"}
Jul 27 19:10:28.181166 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:28.180684    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg" event={"ID":"c6a041af-56a8-423a-aa4a-d70f88eb3d85","Type":"ContainerStarted","Data":"368a94abe8a26c8b8ce8fab55944d627ff6709311275485d88d6ca64e1d6becf"}
Jul 27 19:10:28.200572 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:28.200452    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg" podStartSLOduration=2.200431798 podStartE2EDuration="2.200431798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:10:28.197000202 +0000 UTC m=+6110.241738284" watchObservedRunningTime="2026-07-27 19:10:28.200431798 +0000 UTC m=+6110.245169879"
Jul 27 19:10:29.033360 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:29.033322    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:37.033906 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:37.033810    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:37.046273 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:37.046237    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:10:37.235351 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:10:37.235319    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:11:25.946913 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:25.946875    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"]
Jul 27 19:11:25.956851 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:11:25.956811    2571 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 19:11:25.956851 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 19:11:25.956851 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 19:11:25.956851 ip-10-0-131-74 kubenswrapper[2571]:  > containerID="bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b" cmd=["/bin/sleep","15"]
Jul 27 19:11:25.957187 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:25.956874    2571 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg" message="" err=<
Jul 27 19:11:25.957187 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 19:11:25.957187 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 19:11:25.957187 ip-10-0-131-74 kubenswrapper[2571]:  >
Jul 27 19:11:25.957187 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:11:25.956890    2571 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 19:11:25.957187 ip-10-0-131-74 kubenswrapper[2571]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 19:11:25.957187 ip-10-0-131-74 kubenswrapper[2571]:         , stdout: , stderr: , exit code -1
Jul 27 19:11:25.957187 ip-10-0-131-74 kubenswrapper[2571]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg" podUID="c6a041af-56a8-423a-aa4a-d70f88eb3d85" containerName="main" containerID="cri-o://bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b"
Jul 27 19:11:25.957187 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:25.956937    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg" podUID="c6a041af-56a8-423a-aa4a-d70f88eb3d85" containerName="main" containerID="cri-o://bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b" gracePeriod=60
Jul 27 19:11:26.223827 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.223783    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg" podUID="c6a041af-56a8-423a-aa4a-d70f88eb3d85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 27 19:11:26.305767 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.305741    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:11:26.436575 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.436534    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-home\") pod \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") "
Jul 27 19:11:26.436575 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.436578    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fszt\" (UniqueName: \"kubernetes.io/projected/c6a041af-56a8-423a-aa4a-d70f88eb3d85-kube-api-access-9fszt\") pod \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") "
Jul 27 19:11:26.436843 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.436605    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-tmp-dir\") pod \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") "
Jul 27 19:11:26.436843 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.436636    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-model-cache\") pod \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") "
Jul 27 19:11:26.436843 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.436665    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a041af-56a8-423a-aa4a-d70f88eb3d85-tls-certs\") pod \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") "
Jul 27 19:11:26.436843 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.436711    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-dshm\") pod \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\" (UID: \"c6a041af-56a8-423a-aa4a-d70f88eb3d85\") "
Jul 27 19:11:26.437052 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.436892    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-home" (OuterVolumeSpecName: "home") pod "c6a041af-56a8-423a-aa4a-d70f88eb3d85" (UID: "c6a041af-56a8-423a-aa4a-d70f88eb3d85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:26.437052 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.436966    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6a041af-56a8-423a-aa4a-d70f88eb3d85" (UID: "c6a041af-56a8-423a-aa4a-d70f88eb3d85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:26.437052 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.436984    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-home\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:11:26.437177 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.437095    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6a041af-56a8-423a-aa4a-d70f88eb3d85" (UID: "c6a041af-56a8-423a-aa4a-d70f88eb3d85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:26.439093 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.439067    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-dshm" (OuterVolumeSpecName: "dshm") pod "c6a041af-56a8-423a-aa4a-d70f88eb3d85" (UID: "c6a041af-56a8-423a-aa4a-d70f88eb3d85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:26.439276 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.439256    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a041af-56a8-423a-aa4a-d70f88eb3d85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6a041af-56a8-423a-aa4a-d70f88eb3d85" (UID: "c6a041af-56a8-423a-aa4a-d70f88eb3d85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:11:26.439338 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.439254    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a041af-56a8-423a-aa4a-d70f88eb3d85-kube-api-access-9fszt" (OuterVolumeSpecName: "kube-api-access-9fszt") pod "c6a041af-56a8-423a-aa4a-d70f88eb3d85" (UID: "c6a041af-56a8-423a-aa4a-d70f88eb3d85"). InnerVolumeSpecName "kube-api-access-9fszt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:11:26.469256 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.469226    2571 generic.go:356] "Generic (PLEG): container finished" podID="c6a041af-56a8-423a-aa4a-d70f88eb3d85" containerID="bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b" exitCode=0
Jul 27 19:11:26.469391 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.469271    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"
Jul 27 19:11:26.469391 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.469311    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg" event={"ID":"c6a041af-56a8-423a-aa4a-d70f88eb3d85","Type":"ContainerDied","Data":"bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b"}
Jul 27 19:11:26.469391 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.469370    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg" event={"ID":"c6a041af-56a8-423a-aa4a-d70f88eb3d85","Type":"ContainerDied","Data":"368a94abe8a26c8b8ce8fab55944d627ff6709311275485d88d6ca64e1d6becf"}
Jul 27 19:11:26.469579 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.469400    2571 scope.go:117] "RemoveContainer" containerID="bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b"
Jul 27 19:11:26.478715 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.478694    2571 scope.go:117] "RemoveContainer" containerID="bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b"
Jul 27 19:11:26.478963 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:11:26.478943    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b\": container with ID starting with bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b not found: ID does not exist" containerID="bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b"
Jul 27 19:11:26.479019 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.478970    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b"} err="failed to get container status \"bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b\": rpc error: code = NotFound desc = could not find container \"bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b\": container with ID starting with bdd5fdeec2b5770d9f74bf90c4fb3f8ca679b8d2344e2baeb3fdea0fd866d53b not found: ID does not exist"
Jul 27 19:11:26.493038 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.493016    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"]
Jul 27 19:11:26.494876 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.494855    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qf7xg"]
Jul 27 19:11:26.537890 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.537861    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fszt\" (UniqueName: \"kubernetes.io/projected/c6a041af-56a8-423a-aa4a-d70f88eb3d85-kube-api-access-9fszt\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:11:26.537890 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.537885    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-tmp-dir\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:11:26.538048 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.537895    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-model-cache\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:11:26.538048 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.537905    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a041af-56a8-423a-aa4a-d70f88eb3d85-tls-certs\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:11:26.538048 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:26.537913    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a041af-56a8-423a-aa4a-d70f88eb3d85-dshm\") on node \"ip-10-0-131-74.ec2.internal\" DevicePath \"\""
Jul 27 19:11:28.398428 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:28.398392    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a041af-56a8-423a-aa4a-d70f88eb3d85" path="/var/lib/kubelet/pods/c6a041af-56a8-423a-aa4a-d70f88eb3d85/volumes"
Jul 27 19:11:44.962269 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:11:44.962218    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 19:11:45.556255 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:11:45.556220    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 19:13:48.557411 ip-10-0-131-74 kubenswrapper[2571]: E0727 19:13:48.557310    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6" podUID="fd309d91-1411-4b4b-bedb-3ae7c53a796b"
Jul 27 19:13:49.142154 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:13:49.142116    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6"
Jul 27 19:14:14.113047 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:14.113011    2571 ???:1] "http: TLS handshake error from 10.0.131.74:40136: EOF"
Jul 27 19:14:14.114233 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:14.114211    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/0.log"
Jul 27 19:14:14.136984 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:14.136956    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/1.log"
Jul 27 19:14:15.184658 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:15.184627    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/0.log"
Jul 27 19:14:15.206039 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:15.205996    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/1.log"
Jul 27 19:14:16.280934 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:16.280895    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/0.log"
Jul 27 19:14:16.301560 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:16.301530    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/1.log"
Jul 27 19:14:17.517463 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:17.517428    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/0.log"
Jul 27 19:14:17.540604 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:17.540574    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/1.log"
Jul 27 19:14:39.150021 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:39.149988    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8kxfj_cbe546c0-bfc4-4692-a40b-3f0d22c0760d/global-pull-secret-syncer/0.log"
Jul 27 19:14:39.396360 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:39.396325    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cq6fr_7a89ad7b-c8a0-411a-86b2-d5f70e7c00fa/konnectivity-agent/0.log"
Jul 27 19:14:39.535027 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:39.534988    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-74.ec2.internal_f6587cb3b426ccd7ab481b25ab3a2091/haproxy/0.log"
Jul 27 19:14:46.920761 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:46.920679    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p64pw_b1aa6edb-029a-49bb-8a2a-41e00c595b7b/node-exporter/0.log"
Jul 27 19:14:46.947466 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:46.947439    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p64pw_b1aa6edb-029a-49bb-8a2a-41e00c595b7b/kube-rbac-proxy/0.log"
Jul 27 19:14:46.966419 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:46.966397    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p64pw_b1aa6edb-029a-49bb-8a2a-41e00c595b7b/init-textfile/0.log"
Jul 27 19:14:47.268195 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:47.268160    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-79npr_ba12ae86-da35-4e87-8fce-99b209e13ee8/prometheus-operator/0.log"
Jul 27 19:14:47.285399 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:47.285374    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-79npr_ba12ae86-da35-4e87-8fce-99b209e13ee8/kube-rbac-proxy/0.log"
Jul 27 19:14:47.409366 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:47.409335    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f67999996-4qfvx_35036ce7-166a-4289-b4c2-73b08e3c282e/thanos-query/0.log"
Jul 27 19:14:47.429280 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:47.429257    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f67999996-4qfvx_35036ce7-166a-4289-b4c2-73b08e3c282e/kube-rbac-proxy-web/0.log"
Jul 27 19:14:47.447868 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:47.447845    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f67999996-4qfvx_35036ce7-166a-4289-b4c2-73b08e3c282e/kube-rbac-proxy/0.log"
Jul 27 19:14:47.466977 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:47.466958    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f67999996-4qfvx_35036ce7-166a-4289-b4c2-73b08e3c282e/prom-label-proxy/0.log"
Jul 27 19:14:47.487947 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:47.487919    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f67999996-4qfvx_35036ce7-166a-4289-b4c2-73b08e3c282e/kube-rbac-proxy-rules/0.log"
Jul 27 19:14:47.509555 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:47.509533    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f67999996-4qfvx_35036ce7-166a-4289-b4c2-73b08e3c282e/kube-rbac-proxy-metrics/0.log"
Jul 27 19:14:49.075413 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.075373    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"]
Jul 27 19:14:49.075838 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.075745    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a041af-56a8-423a-aa4a-d70f88eb3d85" containerName="main"
Jul 27 19:14:49.075838 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.075759    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a041af-56a8-423a-aa4a-d70f88eb3d85" containerName="main"
Jul 27 19:14:49.075838 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.075813    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a041af-56a8-423a-aa4a-d70f88eb3d85" containerName="main"
Jul 27 19:14:49.078740 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.078723    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.081262 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.081240    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r8n2x\"/\"kube-root-ca.crt\""
Jul 27 19:14:49.081390 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.081245    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r8n2x\"/\"default-dockercfg-xsd9b\""
Jul 27 19:14:49.082421 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.082380    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r8n2x\"/\"openshift-service-ca.crt\""
Jul 27 19:14:49.087278 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.087256    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"]
Jul 27 19:14:49.206246 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.206208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9c44806-293b-4cc0-b44b-bad6b65e46fd-lib-modules\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.206246 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.206246    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a9c44806-293b-4cc0-b44b-bad6b65e46fd-podres\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.206544 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.206277    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fnnw\" (UniqueName: \"kubernetes.io/projected/a9c44806-293b-4cc0-b44b-bad6b65e46fd-kube-api-access-7fnnw\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.206544 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.206375    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a9c44806-293b-4cc0-b44b-bad6b65e46fd-proc\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.206544 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.206420    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9c44806-293b-4cc0-b44b-bad6b65e46fd-sys\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.307347 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.307306    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9c44806-293b-4cc0-b44b-bad6b65e46fd-sys\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.307559 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.307416    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9c44806-293b-4cc0-b44b-bad6b65e46fd-lib-modules\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.307559 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.307428    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9c44806-293b-4cc0-b44b-bad6b65e46fd-sys\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.307559 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.307444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a9c44806-293b-4cc0-b44b-bad6b65e46fd-podres\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.307559 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.307484    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fnnw\" (UniqueName: \"kubernetes.io/projected/a9c44806-293b-4cc0-b44b-bad6b65e46fd-kube-api-access-7fnnw\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.307559 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.307553    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a9c44806-293b-4cc0-b44b-bad6b65e46fd-proc\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.307775 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.307591    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a9c44806-293b-4cc0-b44b-bad6b65e46fd-podres\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.307775 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.307596    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9c44806-293b-4cc0-b44b-bad6b65e46fd-lib-modules\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.307775 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.307632    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a9c44806-293b-4cc0-b44b-bad6b65e46fd-proc\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.315482 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.315458    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fnnw\" (UniqueName: \"kubernetes.io/projected/a9c44806-293b-4cc0-b44b-bad6b65e46fd-kube-api-access-7fnnw\") pod \"perf-node-gather-daemonset-pfbkd\" (UID: \"a9c44806-293b-4cc0-b44b-bad6b65e46fd\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.389941 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.389854    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:49.517649 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:49.517625    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"]
Jul 27 19:14:49.519550 ip-10-0-131-74 kubenswrapper[2571]: W0727 19:14:49.519525    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda9c44806_293b_4cc0_b44b_bad6b65e46fd.slice/crio-259010143b5cf9714064ac6a598557a6569c3d5c301d47f2d576d55cf8104e08 WatchSource:0}: Error finding container 259010143b5cf9714064ac6a598557a6569c3d5c301d47f2d576d55cf8104e08: Status 404 returned error can't find the container with id 259010143b5cf9714064ac6a598557a6569c3d5c301d47f2d576d55cf8104e08
Jul 27 19:14:50.426906 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:50.426869    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd" event={"ID":"a9c44806-293b-4cc0-b44b-bad6b65e46fd","Type":"ContainerStarted","Data":"e018799f15655e2287313fcb025577f451782e1d36728359d8d88de1ee15732d"}
Jul 27 19:14:50.427381 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:50.426924    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd" event={"ID":"a9c44806-293b-4cc0-b44b-bad6b65e46fd","Type":"ContainerStarted","Data":"259010143b5cf9714064ac6a598557a6569c3d5c301d47f2d576d55cf8104e08"}
Jul 27 19:14:50.427381 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:50.426955    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:50.443309 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:50.443256    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd" podStartSLOduration=1.4432419539999999 podStartE2EDuration="1.443241954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:14:50.441845004 +0000 UTC m=+6372.486583085" watchObservedRunningTime="2026-07-27 19:14:50.443241954 +0000 UTC m=+6372.487980034"
Jul 27 19:14:51.113482 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:51.113449    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zfdj9_b4177ef3-39b7-4328-9299-d1acac07b82d/dns/0.log"
Jul 27 19:14:51.133089 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:51.133058    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zfdj9_b4177ef3-39b7-4328-9299-d1acac07b82d/kube-rbac-proxy/0.log"
Jul 27 19:14:51.153699 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:51.153668    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8lrn8_f50b4a82-db6d-409a-8c83-eb229b3e2e13/dns-node-resolver/0.log"
Jul 27 19:14:51.763672 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:51.763643    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ckvrn_367e4815-f156-4241-88f1-b8e84258fd04/node-ca/0.log"
Jul 27 19:14:53.092224 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:53.092172    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/0.log"
Jul 27 19:14:53.114757 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:53.114729    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/1.log"
Jul 27 19:14:53.646508 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:53.646474    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dvrgq_f6825326-7c3b-4465-bc99-8f541ccad67f/serve-healthcheck-canary/0.log"
Jul 27 19:14:54.277086 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:54.277051    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dd9n4_f514ad0b-436a-4636-a106-63be09beab94/kube-rbac-proxy/0.log"
Jul 27 19:14:54.296652 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:54.296623    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dd9n4_f514ad0b-436a-4636-a106-63be09beab94/exporter/0.log"
Jul 27 19:14:54.321579 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:54.321551    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dd9n4_f514ad0b-436a-4636-a106-63be09beab94/extractor/0.log"
Jul 27 19:14:56.441281 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:56.441254    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-pfbkd"
Jul 27 19:14:57.162376 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:57.162340    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-67d9b7fdbd-hckgz_d32fca4b-75e0-4b3c-a25d-b9352215d749/manager/0.log"
Jul 27 19:14:57.189393 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:57.189368    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-zv4dc_8a752062-406c-4a2d-9768-eff3ce0c2454/openshift-lws-operator/0.log"
Jul 27 19:14:57.841692 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:57.841668    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-ff96d5959-nxnm8_3696de3b-7488-42d9-867e-962e7c2ddcd2/manager/0.log"
Jul 27 19:14:57.899510 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:57.899481    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-5r24t_04f23002-9c6c-4eba-8977-f0e43b81f922/server/0.log"
Jul 27 19:14:59.040851 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:59.040824    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-zggbs_c33e4dc0-1d69-435e-96aa-189134b3d1ab/s3-init/0.log"
Jul 27 19:14:59.059348 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:59.059322    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-zggbs_c33e4dc0-1d69-435e-96aa-189134b3d1ab/download-hf-model/0.log"
Jul 27 19:14:59.094120 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:59.094096    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-q5r7l_dbe2bbc7-a708-4d74-9963-c609e16dc4cc/seaweedfs/0.log"
Jul 27 19:14:59.127646 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:59.127597    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/0.log"
Jul 27 19:14:59.150573 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:14:59.150543    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/1.log"
Jul 27 19:15:00.588086 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:15:00.588059    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/0.log"
Jul 27 19:15:00.611593 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:15:00.611563    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-spfsb_e27bde2a-4e4d-4ea9-9b27-4665b50a751c/istio-proxy/1.log"
Jul 27 19:15:05.489485 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:15:05.489454    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xdrd5_cf8670f0-936b-46cb-af2d-f0465743f16d/kube-multus-additional-cni-plugins/0.log"
Jul 27 19:15:05.511364 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:15:05.511280    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xdrd5_cf8670f0-936b-46cb-af2d-f0465743f16d/egress-router-binary-copy/0.log"
Jul 27 19:15:05.532431 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:15:05.532407    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xdrd5_cf8670f0-936b-46cb-af2d-f0465743f16d/cni-plugins/0.log"
Jul 27 19:15:05.555028 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:15:05.555000    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xdrd5_cf8670f0-936b-46cb-af2d-f0465743f16d/bond-cni-plugin/0.log"
Jul 27 19:15:05.575790 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:15:05.575771    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xdrd5_cf8670f0-936b-46cb-af2d-f0465743f16d/routeoverride-cni/0.log"
Jul 27 19:15:05.601613 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:15:05.601591    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xdrd5_cf8670f0-936b-46cb-af2d-f0465743f16d/whereabouts-cni-bincopy/0.log"
Jul 27 19:15:05.624371 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:15:05.624346    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xdrd5_cf8670f0-936b-46cb-af2d-f0465743f16d/whereabouts-cni/0.log"
Jul 27 19:15:05.703990 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:15:05.703961    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-njw42_7aa19a15-4b11-4bb9-a47a-be6eb5d581db/kube-multus/0.log"
Jul 27 19:15:05.935027 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:15:05.934947    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vtptl_50fb06f3-ab71-43cc-aa9c-58ba1b7651d6/network-metrics-daemon/0.log"
Jul 27 19:15:05.956467 ip-10-0-131-74 kubenswrapper[2571]: I0727 19:15:05.956441    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vtptl_50fb06f3-ab71-43cc-aa9c-58ba1b7651d6/kube-rbac-proxy/0.log"