Jul 31 23:45:57.184212 ip-10-0-131-136 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 23:45:57.626516 ip-10-0-131-136 kubenswrapper[2551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:45:57.626516 ip-10-0-131-136 kubenswrapper[2551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 23:45:57.626516 ip-10-0-131-136 kubenswrapper[2551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:45:57.626516 ip-10-0-131-136 kubenswrapper[2551]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 23:45:57.626516 ip-10-0-131-136 kubenswrapper[2551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:45:57.629241 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.629155    2551 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 23:45:57.632936 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632909    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:57.632936 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632935    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632939    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632943    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632946    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632949    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632952    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632956    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632959    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632962    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632966    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632969    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632971    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632974    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632976    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632979    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632982    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632985    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632987    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632990    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632993    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:57.633003 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632995    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.632998    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633000    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633003    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633006    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633009    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633011    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633014    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633017    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633020    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633023    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633026    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633028    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633031    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633034    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633037    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633039    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633042    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633044    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633047    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:57.633563 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633049    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633052    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633055    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633057    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633060    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633063    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633066    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633069    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633071    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633074    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633077    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633079    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633082    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633085    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633088    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633090    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633093    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633096    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633098    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:57.634062 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633101    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633111    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633117    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633120    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633124    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633127    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633130    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633132    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633135    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633137    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633140    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633142    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633145    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633148    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633150    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633153    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633155    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633158    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633161    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633163    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:57.634530 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633165    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633168    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633172    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633174    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633177    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633179    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633183    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633187    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633192    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633195    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633198    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633201    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633203    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633206    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633209    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633212    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633214    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633217    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633219    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:57.635020 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633931    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633938    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633941    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633944    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633947    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633950    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633953    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633956    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633959    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633962    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633964    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633967    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633970    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633972    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633975    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633978    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633981    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633983    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633986    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633989    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:57.635475 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633992    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633995    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.633998    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634000    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634003    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634006    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634009    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634011    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634013    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634016    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634029    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634033    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634036    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634039    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634043    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634047    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634050    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634054    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:57.636022 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634057    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634059    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634063    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634065    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634069    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634071    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634074    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634077    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634080    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634083    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634086    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634088    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634091    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634094    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634097    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634100    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634102    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634105    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634108    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634110    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:57.636465 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634113    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634115    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634118    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634121    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634123    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634127    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634129    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634132    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634134    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634137    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634139    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634142    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634144    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634147    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634150    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634153    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634155    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634158    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634161    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:57.636967 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634163    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634166    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634169    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634171    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634174    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634177    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634180    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634183    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634186    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634188    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634191    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634193    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634196    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634199    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634201    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634204    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634206    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634209    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634213    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634216    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:57.637440 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634219    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634222    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634592    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634597    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634599    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634602    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634605    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634609    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634614    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634617    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634620    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634623    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634626    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634628    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634631    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634634    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634636    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634639    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634642    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:57.637937 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634645    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634647    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634650    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634653    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634655    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634658    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634661    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634663    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634666    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634669    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634671    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634674    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634676    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634679    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634681    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634684    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634686    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634689    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634691    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634694    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:57.638404 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634696    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634699    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634702    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634707    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634710    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634712    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634715    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634717    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634720    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634723    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634725    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634728    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634731    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634733    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634736    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634739    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634741    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634744    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634747    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634749    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:57.638882 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634752    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634754    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634757    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634759    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634762    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634765    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634767    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634770    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634772    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634775    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634778    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634780    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634783    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634785    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634788    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634793    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634796    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634798    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634801    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634804    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:57.639373 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634806    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634809    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634811    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634814    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634817    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634821    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634825    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634828    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634831    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634834    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634837    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634840    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634842    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634845    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634848    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634851    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634853    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634856    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634859    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:57.639856 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634862    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634865    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.634868    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.634959    2551 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.634972    2551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.634979    2551 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.634983    2551 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.634987    2551 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.634993    2551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.634997    2551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635002    2551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635005    2551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635008    2551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635012    2551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635015    2551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635019    2551 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635022    2551 flags.go:64] FLAG: --cgroup-root=""
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635025    2551 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635028    2551 flags.go:64] FLAG: --client-ca-file=""
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635031    2551 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635034    2551 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635038    2551 flags.go:64] FLAG: --cluster-domain=""
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635041    2551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 23:45:57.640330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635045    2551 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635048    2551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635051    2551 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635055    2551 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635059    2551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635062    2551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635065    2551 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635068    2551 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635071    2551 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635077    2551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635080    2551 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635083    2551 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635087    2551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635090    2551 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635093    2551 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635096    2551 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635099    2551 flags.go:64] FLAG: --enable-server="true"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635103    2551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635108    2551 flags.go:64] FLAG: --event-burst="100"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635111    2551 flags.go:64] FLAG: --event-qps="50"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635114    2551 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635117    2551 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635120    2551 flags.go:64] FLAG: --eviction-hard=""
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635124    2551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 23:45:57.640935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635127    2551 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635130    2551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635133    2551 flags.go:64] FLAG: --eviction-soft=""
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635136    2551 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635139    2551 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635142    2551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635145    2551 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635148    2551 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635151    2551 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635154    2551 flags.go:64] FLAG: --feature-gates=""
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635157    2551 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635160    2551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635163    2551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635166    2551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635169    2551 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635172    2551 flags.go:64] FLAG: --help="false"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635175    2551 flags.go:64] FLAG: --hostname-override="ip-10-0-131-136.ec2.internal"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635180    2551 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635183    2551 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635186    2551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635189    2551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635192    2551 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635195    2551 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 23:45:57.641519 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635198    2551 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635201    2551 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635205    2551 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635208    2551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635211    2551 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635221    2551 flags.go:64] FLAG: --kube-reserved=""
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635224    2551 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635227    2551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635230    2551 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635233    2551 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635236    2551 flags.go:64] FLAG: --lock-file=""
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635239    2551 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635242    2551 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635245    2551 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635250    2551 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635253    2551 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635256    2551 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635259    2551 flags.go:64] FLAG: --logging-format="text"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635262    2551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635265    2551 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635268    2551 flags.go:64] FLAG: --manifest-url=""
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635271    2551 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635275    2551 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635278    2551 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635282    2551 flags.go:64] FLAG: --max-pods="110"
Jul 31 23:45:57.642129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635285    2551 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635289    2551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635292    2551 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635295    2551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635298    2551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635301    2551 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635304    2551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635311    2551 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635313    2551 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635319    2551 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635322    2551 flags.go:64] FLAG: --pod-cidr=""
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635324    2551 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635330    2551 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635340    2551 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635343    2551 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635346    2551 flags.go:64] FLAG: --port="10250"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635349    2551 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635352    2551 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-048b645a90149d25f"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635355    2551 flags.go:64] FLAG: --qos-reserved=""
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635358    2551 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635361    2551 flags.go:64] FLAG: --register-node="true"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635364    2551 flags.go:64] FLAG: --register-with-taints=""
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635368    2551 flags.go:64] FLAG: --registry-burst="10"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635371    2551 flags.go:64] FLAG: --registry-qps="5"
Jul 31 23:45:57.642811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635373    2551 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635376    2551 flags.go:64] FLAG: --reserved-memory=""
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635380    2551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635383    2551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635385    2551 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635388    2551 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635391    2551 flags.go:64] FLAG: --runonce="false"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635394    2551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635397    2551 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635401    2551 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635404    2551 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635407    2551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635410    2551 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635413    2551 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635416    2551 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635419    2551 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635422    2551 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635425    2551 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635430    2551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635433    2551 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635436    2551 flags.go:64] FLAG: --system-cgroups=""
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635439    2551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635450    2551 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635453    2551 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635456    2551 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 23:45:57.643765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635459    2551 flags.go:64] FLAG: --tls-min-version=""
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635462    2551 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635465    2551 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635468    2551 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635471    2551 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635474    2551 flags.go:64] FLAG: --v="2"
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635478    2551 flags.go:64] FLAG: --version="false"
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635482    2551 flags.go:64] FLAG: --vmodule=""
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635486    2551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.635489    2551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635638    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635642    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635645    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635648    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635651    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635654    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635658    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635661    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635663    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635666    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635668    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635671    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:57.644458 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635674    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635676    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635678    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635682    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635685    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635688    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635690    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635693    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635701    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635704    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635706    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635709    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635711    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635714    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635717    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635719    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635722    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635724    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635727    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635729    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:57.645049 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635732    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635734    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635737    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635739    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635742    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635744    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635751    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635754    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635756    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635759    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635761    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635764    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635767    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635769    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635772    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635775    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635778    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635780    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635783    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635786    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:57.645549 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635788    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635796    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635799    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635802    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635804    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635807    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635810    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635812    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635815    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635818    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635820    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635823    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635827    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635830    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635833    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635835    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635838    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635840    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635844    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635846    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:57.646055 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635849    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635852    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635854    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635857    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635859    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635862    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635864    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635869    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635872    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635877    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635880    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635882    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635885    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635888    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635896    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635899    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635902    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635904    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635907    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:57.646553 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635909    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635912    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635926    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635929    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635932    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635934    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635937    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.635941    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.636743    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.644672    2551 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.644686    2551 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.644706    2551 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.644712    2551 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644771    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:57.647043 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644776    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644779    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644782    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644785    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644788    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644791    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644794    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644798    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644801    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644803    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644807    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644810    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644813    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644815    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644819    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644821    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644824    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644827    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644829    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644832    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:57.647396 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644835    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644838    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644840    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644843    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644845    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644848    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644851    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644854    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644856    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644859    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644862    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644865    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644868    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644870    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644873    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644876    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644878    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644881    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644883    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644886    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:57.647875 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644889    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644891    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644894    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644898    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644902    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644906    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644908    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644911    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644928    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644933    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644936    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644939    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644942    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644944    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644949    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644952    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644956    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644958    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644961    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:57.648374 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644964    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644966    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644969    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644972    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644975    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644977    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644980    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644983    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644986    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644988    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644991    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644993    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644996    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.644998    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645001    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645004    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645006    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645009    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645011    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645015    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:57.648838 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645018    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645020    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645023    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645042    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645046    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645049    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645053    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645056    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645059    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645062    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645064    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645067    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645070    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645072    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645075    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645077    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645081    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645085    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:57.649424 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645087    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.645092    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645224    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645230    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645233    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645236    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645238    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645242    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645246    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645250    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645253    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645256    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645259    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645262    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:45:57.649852 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645266    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645270    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645274    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645280    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645284    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645288    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645292    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645295    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645298    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645301    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645303    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645306    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645308    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645311    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645314    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645317    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645319    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645322    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645324    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645327    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:45:57.650214 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645330    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645333    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645335    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645338    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645341    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645343    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645346    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645349    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645351    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645354    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645356    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645359    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645361    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645366    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645369    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645372    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645374    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645377    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645379    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:45:57.650692 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645382    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645386    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645389    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645392    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645395    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645398    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645401    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645403    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645406    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645408    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645411    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645414    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645417    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645420    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645423    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645425    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645428    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645431    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645433    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645436    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:45:57.651200 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645438    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645441    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645444    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645446    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645449    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645452    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645454    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645458    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645461    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645463    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645465    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645468    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645471    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645473    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645475    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645478    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645480    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645483    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645485    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645488    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:45:57.651676 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645490    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:45:57.652178 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645495    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:45:57.652178 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645497    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:45:57.652178 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645500    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:45:57.652178 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645503    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:45:57.652178 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645505    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:45:57.652178 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645508    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:45:57.652178 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:45:57.645510    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:45:57.652178 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.645515    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:45:57.652178 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.645642    2551 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 23:45:57.652178 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.648238    2551 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 23:45:57.652178 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.649815    2551 server.go:1019] "Starting client certificate rotation"
Jul 31 23:45:57.652178 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.649953    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:45:57.652178 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.650001    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:45:57.652792 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.652779    2551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:45:57.654412 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.654398    2551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:45:57.666585 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.666564    2551 log.go:25] "Validated CRI v1 runtime API"
Jul 31 23:45:57.668854 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.668834    2551 log.go:25] "Validated CRI v1 image API"
Jul 31 23:45:57.670478 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.670456    2551 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 23:45:57.676597 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.676570    2551 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8e9b1797-73f3-402c-9db0-45c1d6eb1acc:/dev/nvme0n1p4 d3f4d3c8-29a8-4176-98f6-6e69713ae078:/dev/nvme0n1p3]
Jul 31 23:45:57.676666 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.676595    2551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 23:45:57.678940 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.678910    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:45:57.683289 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.682960    2551 manager.go:216] Machine: {Timestamp:2026-07-31 23:45:57.680160695 +0000 UTC m=+0.374208101 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3240591 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec236a19ab12d1eee6ae651e8be87c79 SystemUUID:ec236a19-ab12-d1ee-e6ae-651e8be87c79 BootID:fc4d5a54-85c4-42f3-9643-e11f2e44aceb Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f0:72:bd:a9:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f0:72:bd:a9:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:2a:fa:d0:86:34 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 23:45:57.683289 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.683219    2551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 23:45:57.683416 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.683338    2551 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 23:45:57.685531 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.685500    2551 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 23:45:57.685658 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.685532    2551 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-136.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 23:45:57.686405 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.686394    2551 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 23:45:57.686447 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.686406    2551 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 23:45:57.686447 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.686414    2551 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:45:57.687099 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.687088    2551 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:45:57.687898 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.687889    2551 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 23:45:57.692414 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.692402    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:45:57.692513 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.692504    2551 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 23:45:57.692550 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.692531    2551 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 23:45:57.692550 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.692540    2551 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 23:45:57.692611 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.692554    2551 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 23:45:57.692611 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.692563    2551 kubelet.go:398] "Adding apiserver pod source"
Jul 31 23:45:57.692611 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.692572    2551 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 23:45:57.692902 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.692889    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:45:57.692958 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.692928    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:45:57.695206 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.695190    2551 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 23:45:57.696874 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.696861    2551 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 23:45:57.696928 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.696891    2551 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 23:45:57.697910 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.697898    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 23:45:57.697978 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.697928    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 23:45:57.697978 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.697936    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 23:45:57.697978 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.697953    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 23:45:57.697978 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.697967    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 23:45:57.697978 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.697976    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 23:45:57.698128 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.697988    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 23:45:57.698128 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.697998    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 23:45:57.698128 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.698007    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 23:45:57.698128 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.698017    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 23:45:57.698128 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.698027    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 23:45:57.698128 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.698033    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 23:45:57.700102 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.700085    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 23:45:57.700138 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.700105    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 23:45:57.701737 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.701710    2551 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 23:45:57.701907 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.701888    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 23:45:57.701988 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.701963    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 23:45:57.703430 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.703418    2551 server.go:1268] "Started kubelet"
Jul 31 23:45:57.703567 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.703495    2551 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 23:45:57.703664 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.703510    2551 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 23:45:57.703664 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.703611    2551 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 23:45:57.704201 ip-10-0-131-136 systemd[1]: Started Kubernetes Kubelet.
Jul 31 23:45:57.704729 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.704710    2551 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 23:45:57.705787 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.705773    2551 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 23:45:57.710797 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.710779    2551 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 23:45:57.710891 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.710833    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 23:45:57.711546 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.711533    2551 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 23:45:57.711687 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.711666    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found"
Jul 31 23:45:57.712022 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.712006    2551 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 23:45:57.712084 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.712024    2551 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 23:45:57.712134 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.712007    2551 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 23:45:57.712178 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.712149    2551 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 23:45:57.712178 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.712161    2551 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 23:45:57.712391 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.712377    2551 factory.go:55] Registering systemd factory
Jul 31 23:45:57.712446 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.712395    2551 factory.go:223] Registration of the systemd container factory successfully
Jul 31 23:45:57.712617 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.712604    2551 factory.go:153] Registering CRI-O factory
Jul 31 23:45:57.712675 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.712619    2551 factory.go:223] Registration of the crio container factory successfully
Jul 31 23:45:57.712675 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.712660    2551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 23:45:57.712772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.712686    2551 factory.go:103] Registering Raw factory
Jul 31 23:45:57.712772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.712697    2551 manager.go:1196] Started watching for new ooms in manager
Jul 31 23:45:57.713181 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.713158    2551 manager.go:349] Starting recovery of all containers
Jul 31 23:45:57.714191 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.714170    2551 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 23:45:57.714284 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.714273    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 23:45:57.714622 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.713569    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18c78496269bae55  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:ip-10-0-131-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-07-31 23:45:57.703388757 +0000 UTC m=+0.397436165,LastTimestamp:2026-07-31 23:45:57.703388757 +0000 UTC m=+0.397436165,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}"
Jul 31 23:45:57.725730 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.725713    2551 manager.go:354] Recovery completed
Jul 31 23:45:57.729878 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.729864    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:57.733982 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.733951    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:57.734065 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.733997    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:57.734065 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.734009    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:57.735127 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.735110    2551 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 23:45:57.735127 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.735121    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:57.735127 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.735125    2551 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 23:45:57.735304 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.735173    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:45:57.735936 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.735906    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:57.736034 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.735962    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:57.736034 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.735977    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:57.736370 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.736356    2551 policy_none.go:49] "None policy: Start"
Jul 31 23:45:57.736424 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.736378    2551 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 23:45:57.736424 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.736397    2551 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 23:45:57.737491 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.737478    2551 policy_none.go:47] "Start"
Jul 31 23:45:57.745888 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.745687    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18c78496286e79a0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:ip-10-0-131-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-07-31 23:45:57.733980576 +0000 UTC m=+0.428027983,LastTimestamp:2026-07-31 23:45:57.733980576 +0000 UTC m=+0.428027983,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}"
Jul 31 23:45:57.750215 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.750197    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tmnfg"
Jul 31 23:45:57.754643 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.754561    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18c78496286ed37c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:ip-10-0-131-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-07-31 23:45:57.73400358 +0000 UTC m=+0.428050986,LastTimestamp:2026-07-31 23:45:57.73400358 +0000 UTC m=+0.428050986,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}"
Jul 31 23:45:57.759997 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.759978    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tmnfg"
Jul 31 23:45:57.764447 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.764393    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18c78496286efb7c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:ip-10-0-131-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-07-31 23:45:57.73401382 +0000 UTC m=+0.428061232,LastTimestamp:2026-07-31 23:45:57.73401382 +0000 UTC m=+0.428061232,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}"
Jul 31 23:45:57.769522 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.769508    2551 manager.go:337] "Starting Device Plugin manager"
Jul 31 23:45:57.769583 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.769539    2551 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 23:45:57.769583 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.769550    2551 server.go:87] "Starting device plugin registration server"
Jul 31 23:45:57.769784 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.769772    2551 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 23:45:57.769841 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.769786    2551 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 23:45:57.769901 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.769880    2551 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 23:45:57.769982 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.769969    2551 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 23:45:57.769982 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.769982    2551 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 23:45:57.770498 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.770478    2551 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 23:45:57.770574 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.770525    2551 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-136.ec2.internal\" not found"
Jul 31 23:45:57.776320 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.776254    2551 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18c78496286e79a0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18c78496286e79a0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:ip-10-0-131-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-07-31 23:45:57.733980576 +0000 UTC m=+0.428027983,LastTimestamp:2026-07-31 23:45:57.735945288 +0000 UTC m=+0.429992700,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}"
Jul 31 23:45:57.785656 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.785596    2551 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18c78496286ed37c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18c78496286ed37c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:ip-10-0-131-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-07-31 23:45:57.73400358 +0000 UTC m=+0.428050986,LastTimestamp:2026-07-31 23:45:57.735969688 +0000 UTC m=+0.430017096,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}"
Jul 31 23:45:57.795577 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.795521    2551 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18c78496286efb7c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18c78496286efb7c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:ip-10-0-131-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-07-31 23:45:57.73401382 +0000 UTC m=+0.428061232,LastTimestamp:2026-07-31 23:45:57.735982725 +0000 UTC m=+0.430030138,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}"
Jul 31 23:45:57.806363 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.806303    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18c784962aae14ce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:ip-10-0-131-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-07-31 23:45:57.771703502 +0000 UTC m=+0.465750896,LastTimestamp:2026-07-31 23:45:57.771703502 +0000 UTC m=+0.465750896,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}"
Jul 31 23:45:57.870668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.870631    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:57.871665 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.871648    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:57.871754 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.871679    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:57.871754 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.871693    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:57.871754 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.871725    2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-136.ec2.internal"
Jul 31 23:45:57.879451 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.879336    2551 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18c78496286e79a0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18c78496286e79a0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:ip-10-0-131-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-07-31 23:45:57.733980576 +0000 UTC m=+0.428027983,LastTimestamp:2026-07-31 23:45:57.871663869 +0000 UTC m=+0.565711280,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}"
Jul 31 23:45:57.881881 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.881862    2551 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-136.ec2.internal"
Jul 31 23:45:57.881965 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.881830    2551 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18c78496286ed37c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18c78496286ed37c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:ip-10-0-131-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-07-31 23:45:57.73400358 +0000 UTC m=+0.428050986,LastTimestamp:2026-07-31 23:45:57.871686033 +0000 UTC m=+0.565733440,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}"
Jul 31 23:45:57.889195 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.889128    2551 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-136.ec2.internal.18c78496286efb7c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-136.ec2.internal.18c78496286efb7c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-136.ec2.internal,UID:ip-10-0-131-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-136.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-136.ec2.internal,},FirstTimestamp:2026-07-31 23:45:57.73401382 +0000 UTC m=+0.428061232,LastTimestamp:2026-07-31 23:45:57.871698383 +0000 UTC m=+0.565745791,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-136.ec2.internal,}"
Jul 31 23:45:57.920853 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.920829    2551 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-136.ec2.internal\" not found" node="ip-10-0-131-136.ec2.internal"
Jul 31 23:45:57.939534 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.939489    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 23:45:57.940703 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.940687    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 23:45:57.940772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.940727    2551 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 23:45:57.940772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.940748    2551 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 23:45:57.940877 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:57.940794    2551 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 23:45:57.943095 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:57.943077    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:45:58.041334 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.041289    2551 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal"]
Jul 31 23:45:58.041483 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.041373    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:58.042325 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.042309    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:58.042400 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.042339    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:58.042400 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.042348    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:58.043603 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.043589    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:58.043742 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.043722    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.043804 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.043798    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:58.044340 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.044325    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:58.044425 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.044346    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:58.044425 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.044355    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:58.044425 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.044356    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:58.044425 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.044379    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:58.044425 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.044393    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:58.045543 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.045525    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.045626 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.045553    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:58.046202 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.046185    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:58.046265 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.046218    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:58.046265 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.046234    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:58.073451 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:58.073428    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.ec2.internal\" not found" node="ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.077650 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:58.077635    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-136.ec2.internal\" not found" node="ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.082171 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.082159    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:45:58.083186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.083172    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:45:58.083254 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.083196    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:45:58.083254 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.083205    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:45:58.083254 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.083225    2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.094538 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.094494    2551 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.094788 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:58.094774    2551 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-136.ec2.internal\": node \"ip-10-0-131-136.ec2.internal\" not found"
Jul 31 23:45:58.107271 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:58.107255    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found"
Jul 31 23:45:58.113478 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.113456    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d4efdb1fd75703bb7b34b57a3634782a-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.ec2.internal\" (UID: \"d4efdb1fd75703bb7b34b57a3634782a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.113538 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.113481    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41e595559883d998bfb59f4d787bbea5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"41e595559883d998bfb59f4d787bbea5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.113538 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.113498    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41e595559883d998bfb59f4d787bbea5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"41e595559883d998bfb59f4d787bbea5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.207677 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:58.207611    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found"
Jul 31 23:45:58.213914 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.213894    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d4efdb1fd75703bb7b34b57a3634782a-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.ec2.internal\" (UID: \"d4efdb1fd75703bb7b34b57a3634782a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.213984 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.213932    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d4efdb1fd75703bb7b34b57a3634782a-config\") pod \"kube-apiserver-proxy-ip-10-0-131-136.ec2.internal\" (UID: \"d4efdb1fd75703bb7b34b57a3634782a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.213984 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.213945    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41e595559883d998bfb59f4d787bbea5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"41e595559883d998bfb59f4d787bbea5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.213984 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.213964    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41e595559883d998bfb59f4d787bbea5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"41e595559883d998bfb59f4d787bbea5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.214090 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.213992    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41e595559883d998bfb59f4d787bbea5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"41e595559883d998bfb59f4d787bbea5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.214090 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.214000    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41e595559883d998bfb59f4d787bbea5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal\" (UID: \"41e595559883d998bfb59f4d787bbea5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.308084 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:58.308050    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found"
Jul 31 23:45:58.376551 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.376521    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.381056 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.380644    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.408473 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:58.408449    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found"
Jul 31 23:45:58.510176 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:58.510131    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found"
Jul 31 23:45:58.610273 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:58.610239    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found"
Jul 31 23:45:58.633675 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.633647    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:45:58.648630 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.648590    2551 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 23:45:58.648808 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.648782    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 23:45:58.648850 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.648794    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 23:45:58.710480 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:58.710454    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-136.ec2.internal\" not found"
Jul 31 23:45:58.749099 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.745739    2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:45:58.811753 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.811724    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.824107 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.824071    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:45:58.826154 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.826139    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal"
Jul 31 23:45:58.835096 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.835071    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:45:58.887257 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.887210    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 23:40:57 +0000 UTC" deadline="2028-01-15 06:54:29.256024351 +0000 UTC"
Jul 31 23:45:58.887257 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.887257    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12775h8m30.368770838s"
Jul 31 23:45:58.954868 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:58.954849    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:45:59.034731 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.034664    2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:45:59.692881 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.692839    2551 apiserver.go:52] "Watching apiserver"
Jul 31 23:45:59.701019 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.700976    2551 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 23:45:59.701569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.701541    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-jqkv8","openshift-ovn-kubernetes/ovnkube-node-b47km","kube-system/konnectivity-agent-5z6n7","kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9","openshift-cluster-node-tuning-operator/tuned-9ppkm","openshift-dns/node-resolver-k7f2q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal","openshift-multus/network-metrics-daemon-r7fbk","openshift-network-diagnostics/network-check-target-j2rjc","kube-system/global-pull-secret-syncer-hfz7q","openshift-image-registry/node-ca-wkzlt","openshift-multus/multus-86226","openshift-multus/multus-additional-cni-plugins-66n9v"]
Jul 31 23:45:59.703768 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.703746    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jqkv8"
Jul 31 23:45:59.706122 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.705661    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.710020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.706959    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hwbp8\""
Jul 31 23:45:59.710020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.707332    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 23:45:59.710020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.707527    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:45:59.710020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.707736    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 23:45:59.710020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.709606    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 23:45:59.710020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.709791    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 23:45:59.710020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.709801    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 23:45:59.710020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.709945    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5z6n7"
Jul 31 23:45:59.710020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.710014    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 23:45:59.710527 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.710290    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.710527 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.710486    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.711667 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.711535    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k7f2q"
Jul 31 23:45:59.718602 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.718109    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2pkbp\""
Jul 31 23:45:59.718602 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.718385    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 23:45:59.720472 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.719377    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 23:45:59.720472 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.719604    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 23:45:59.720472 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.719708    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 23:45:59.720472 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.719904    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x97fg\""
Jul 31 23:45:59.720472 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.719917    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 23:45:59.720472 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.720110    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 23:45:59.720472 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.720139    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-299z5\""
Jul 31 23:45:59.720472 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.720313    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 23:45:59.720472 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.720323    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5cdj6\""
Jul 31 23:45:59.721462 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.720678    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:45:59.721462 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.720810    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 23:45:59.721462 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:59.720815    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:45:59.722874 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.721713    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 23:45:59.722874 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.722288    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:45:59.722874 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:59.722393    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:45:59.723383 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.723112    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 23:45:59.723954 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.723445    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sktv6\""
Jul 31 23:45:59.724966 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.724388    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:45:59.724966 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.724674    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:45:59.724966 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:59.724743    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.726706    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvd9\" (UniqueName: \"kubernetes.io/projected/b96b8478-7682-4a69-affb-0fc5239b5a4e-kube-api-access-2pvd9\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.726746    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-sysctl-d\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.726771    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-sysctl-conf\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.726799    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs\") pod \"network-metrics-daemon-r7fbk\" (UID: \"a0d57ecc-0561-4cce-855e-0281fb270d89\") " pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.726855    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-cni-netd\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.726883    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818c8b69-f230-4bf7-b050-ae9e23c0281b-tmp-dir\") pod \"node-resolver-k7f2q\" (UID: \"818c8b69-f230-4bf7-b050-ae9e23c0281b\") " pod="openshift-dns/node-resolver-k7f2q"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.726909    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hdfg\" (UniqueName: \"kubernetes.io/projected/a0d57ecc-0561-4cce-855e-0281fb270d89-kube-api-access-4hdfg\") pod \"network-metrics-daemon-r7fbk\" (UID: \"a0d57ecc-0561-4cce-855e-0281fb270d89\") " pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.726933    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be479e0b-db97-4752-9f8a-4676b12319c5-konnectivity-ca\") pod \"konnectivity-agent-5z6n7\" (UID: \"be479e0b-db97-4752-9f8a-4676b12319c5\") " pod="kube-system/konnectivity-agent-5z6n7"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.726959    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-registration-dir\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.726986    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-sys\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727110    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-run-ovn\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727181    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-cni-bin\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727307    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84c96591-3b96-4722-a376-ea2b0728b358-ovn-node-metrics-cert\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727332    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-lib-modules\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727358    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-var-lib-kubelet\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.727654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727385    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-tmp\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727409    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99ctn\" (UniqueName: \"kubernetes.io/projected/818c8b69-f230-4bf7-b050-ae9e23c0281b-kube-api-access-99ctn\") pod \"node-resolver-k7f2q\" (UID: \"818c8b69-f230-4bf7-b050-ae9e23c0281b\") " pod="openshift-dns/node-resolver-k7f2q"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727449    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4034a0ac-d7a3-4689-8f4b-2281d0d0722a-iptables-alerter-script\") pod \"iptables-alerter-jqkv8\" (UID: \"4034a0ac-d7a3-4689-8f4b-2281d0d0722a\") " pod="openshift-network-operator/iptables-alerter-jqkv8"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727476    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4034a0ac-d7a3-4689-8f4b-2281d0d0722a-host-slash\") pod \"iptables-alerter-jqkv8\" (UID: \"4034a0ac-d7a3-4689-8f4b-2281d0d0722a\") " pod="openshift-network-operator/iptables-alerter-jqkv8"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727501    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-kubelet\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727547    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84c96591-3b96-4722-a376-ea2b0728b358-env-overrides\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727581    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84c96591-3b96-4722-a376-ea2b0728b358-ovnkube-script-lib\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727606    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727630    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-socket-dir\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727654    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5tb2\" (UniqueName: \"kubernetes.io/projected/4034a0ac-d7a3-4689-8f4b-2281d0d0722a-kube-api-access-l5tb2\") pod \"iptables-alerter-jqkv8\" (UID: \"4034a0ac-d7a3-4689-8f4b-2281d0d0722a\") " pod="openshift-network-operator/iptables-alerter-jqkv8"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727678    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-run-netns\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727702    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-node-log\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727723    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-log-socket\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727747    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727773    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-sys-fs\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727797    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-modprobe-d\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.728511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727820    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-kubernetes\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727846    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-slash\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727895    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-device-dir\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727920    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-sysconfig\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727945    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-systemd\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727968    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-run\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.727990    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-tuned\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.728037    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-run-systemd\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.728061    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-var-lib-openvswitch\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.728089    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-etc-openvswitch\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.728113    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-run-ovn-kubernetes\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.728137    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be479e0b-db97-4752-9f8a-4676b12319c5-agent-certs\") pod \"konnectivity-agent-5z6n7\" (UID: \"be479e0b-db97-4752-9f8a-4676b12319c5\") " pod="kube-system/konnectivity-agent-5z6n7"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.728162    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.728185    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-host\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.728211    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn5b6\" (UniqueName: \"kubernetes.io/projected/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-kube-api-access-xn5b6\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.728234    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/818c8b69-f230-4bf7-b050-ae9e23c0281b-hosts-file\") pod \"node-resolver-k7f2q\" (UID: \"818c8b69-f230-4bf7-b050-ae9e23c0281b\") " pod="openshift-dns/node-resolver-k7f2q"
Jul 31 23:45:59.729328 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.728257    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-systemd-units\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.730139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.728285    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-run-openvswitch\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.730139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.728312    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84c96591-3b96-4722-a376-ea2b0728b358-ovnkube-config\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.730139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.728336    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz6jr\" (UniqueName: \"kubernetes.io/projected/84c96591-3b96-4722-a376-ea2b0728b358-kube-api-access-pz6jr\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.730139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.729705    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 23:45:59.732404 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.731944    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wkzlt"
Jul 31 23:45:59.733493 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.733312    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86226"
Jul 31 23:45:59.734424 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.734403    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 23:45:59.734529 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.734421    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-84w28\""
Jul 31 23:45:59.734675 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.734657    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 23:45:59.734805 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.734754    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.734869 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.734757    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 23:45:59.735674 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.735593    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 23:45:59.735872 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.735853    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9q92m\""
Jul 31 23:45:59.736078 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.735876    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 23:45:59.736271 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.736248    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 23:45:59.736418 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.736263    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 23:45:59.738118 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.737929    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 23:45:59.738200 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.738152    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 23:45:59.738416 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.738395    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vfmzb\""
Jul 31 23:45:59.741414 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.741393    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:45:59.767363 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.767342    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vmpzd"
Jul 31 23:45:59.776178 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.776155    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vmpzd"
Jul 31 23:45:59.814027 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.813993    2551 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 23:45:59.829230 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829198    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61b45fe3-b6e6-47f9-8974-11cb808bc9f0-host\") pod \"node-ca-wkzlt\" (UID: \"61b45fe3-b6e6-47f9-8974-11cb808bc9f0\") " pod="openshift-image-registry/node-ca-wkzlt"
Jul 31 23:45:59.829346 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829251    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84c96591-3b96-4722-a376-ea2b0728b358-env-overrides\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.829346 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829283    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84c96591-3b96-4722-a376-ea2b0728b358-ovnkube-script-lib\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.829346 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829302    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.829346 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829322    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-socket-dir\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.829564 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829361    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5tb2\" (UniqueName: \"kubernetes.io/projected/4034a0ac-d7a3-4689-8f4b-2281d0d0722a-kube-api-access-l5tb2\") pod \"iptables-alerter-jqkv8\" (UID: \"4034a0ac-d7a3-4689-8f4b-2281d0d0722a\") " pod="openshift-network-operator/iptables-alerter-jqkv8"
Jul 31 23:45:59.829564 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829376    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-run-netns\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.829564 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829394    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-node-log\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.829564 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829413    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-systemd\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.829564 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829430    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-os-release\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.829564 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829455    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-etc-kubernetes\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.829564 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829470    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/502fbd41-3e97-4d6e-ac72-d6a05311d675-cnibin\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.829564 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829493    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/502fbd41-3e97-4d6e-ac72-d6a05311d675-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.829564 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829513    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-slash\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.829564 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829528    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-run\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.829564 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829542    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-tuned\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.829564 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829556    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-run-netns\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829580    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l9mk\" (UniqueName: \"kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk\") pod \"network-check-target-j2rjc\" (UID: \"15c09ad2-cc88-49b1-b608-6f4bab832087\") " pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829597    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-run-systemd\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829621    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-var-lib-openvswitch\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829645    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-run-ovn-kubernetes\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829681    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be479e0b-db97-4752-9f8a-4676b12319c5-agent-certs\") pod \"konnectivity-agent-5z6n7\" (UID: \"be479e0b-db97-4752-9f8a-4676b12319c5\") " pod="kube-system/konnectivity-agent-5z6n7"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829697    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/818c8b69-f230-4bf7-b050-ae9e23c0281b-hosts-file\") pod \"node-resolver-k7f2q\" (UID: \"818c8b69-f230-4bf7-b050-ae9e23c0281b\") " pod="openshift-dns/node-resolver-k7f2q"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829714    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-systemd-units\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829737    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-run-openvswitch\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829794    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-sysctl-conf\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829824    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-run-multus-certs\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829842    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/502fbd41-3e97-4d6e-ac72-d6a05311d675-cni-binary-copy\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829863    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-cni-netd\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829882    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvcbm\" (UniqueName: \"kubernetes.io/projected/61b45fe3-b6e6-47f9-8974-11cb808bc9f0-kube-api-access-cvcbm\") pod \"node-ca-wkzlt\" (UID: \"61b45fe3-b6e6-47f9-8974-11cb808bc9f0\") " pod="openshift-image-registry/node-ca-wkzlt"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829903    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-var-lib-cni-bin\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829927    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-hostroot\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829951    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw75l\" (UniqueName: \"kubernetes.io/projected/502fbd41-3e97-4d6e-ac72-d6a05311d675-kube-api-access-cw75l\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.830186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.829981    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hdfg\" (UniqueName: \"kubernetes.io/projected/a0d57ecc-0561-4cce-855e-0281fb270d89-kube-api-access-4hdfg\") pod \"network-metrics-daemon-r7fbk\" (UID: \"a0d57ecc-0561-4cce-855e-0281fb270d89\") " pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830029    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be479e0b-db97-4752-9f8a-4676b12319c5-konnectivity-ca\") pod \"konnectivity-agent-5z6n7\" (UID: \"be479e0b-db97-4752-9f8a-4676b12319c5\") " pod="kube-system/konnectivity-agent-5z6n7"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830061    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-sys\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830085    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-cnibin\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830113    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-run-k8s-cni-cncf-io\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830140    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-run-ovn\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830163    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40e0e72a-edb8-4893-9b23-e12962884e30-cni-binary-copy\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830188    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vwb4\" (UniqueName: \"kubernetes.io/projected/40e0e72a-edb8-4893-9b23-e12962884e30-kube-api-access-8vwb4\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830211    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20d38ffc-045d-41cf-bcde-a74d3e0b8361-dbus\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830243    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99ctn\" (UniqueName: \"kubernetes.io/projected/818c8b69-f230-4bf7-b050-ae9e23c0281b-kube-api-access-99ctn\") pod \"node-resolver-k7f2q\" (UID: \"818c8b69-f230-4bf7-b050-ae9e23c0281b\") " pod="openshift-dns/node-resolver-k7f2q"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830270    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4034a0ac-d7a3-4689-8f4b-2281d0d0722a-iptables-alerter-script\") pod \"iptables-alerter-jqkv8\" (UID: \"4034a0ac-d7a3-4689-8f4b-2281d0d0722a\") " pod="openshift-network-operator/iptables-alerter-jqkv8"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830299    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4034a0ac-d7a3-4689-8f4b-2281d0d0722a-host-slash\") pod \"iptables-alerter-jqkv8\" (UID: \"4034a0ac-d7a3-4689-8f4b-2281d0d0722a\") " pod="openshift-network-operator/iptables-alerter-jqkv8"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830328    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-kubelet\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830363    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-modprobe-d\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830392    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-kubernetes\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830432    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-host\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830460    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn5b6\" (UniqueName: \"kubernetes.io/projected/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-kube-api-access-xn5b6\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.831054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830492    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-log-socket\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830531    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830568    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-sys-fs\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830595    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-var-lib-cni-multus\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830621    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/502fbd41-3e97-4d6e-ac72-d6a05311d675-os-release\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830647    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20d38ffc-045d-41cf-bcde-a74d3e0b8361-kubelet-config\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830676    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-device-dir\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830704    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-sysconfig\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830728    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-system-cni-dir\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830752    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-multus-cni-dir\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830780    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-multus-socket-dir-parent\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830808    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/502fbd41-3e97-4d6e-ac72-d6a05311d675-system-cni-dir\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830835    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/502fbd41-3e97-4d6e-ac72-d6a05311d675-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830861    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61b45fe3-b6e6-47f9-8974-11cb808bc9f0-serviceca\") pod \"node-ca-wkzlt\" (UID: \"61b45fe3-b6e6-47f9-8974-11cb808bc9f0\") " pod="openshift-image-registry/node-ca-wkzlt"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830891    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-etc-openvswitch\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830919    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.831905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830946    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40e0e72a-edb8-4893-9b23-e12962884e30-multus-daemon-config\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.830973    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831020    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84c96591-3b96-4722-a376-ea2b0728b358-ovnkube-config\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831049    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz6jr\" (UniqueName: \"kubernetes.io/projected/84c96591-3b96-4722-a376-ea2b0728b358-kube-api-access-pz6jr\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831078    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvd9\" (UniqueName: \"kubernetes.io/projected/b96b8478-7682-4a69-affb-0fc5239b5a4e-kube-api-access-2pvd9\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831106    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-sysctl-d\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831135    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/502fbd41-3e97-4d6e-ac72-d6a05311d675-tuning-conf-dir\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831169    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs\") pod \"network-metrics-daemon-r7fbk\" (UID: \"a0d57ecc-0561-4cce-855e-0281fb270d89\") " pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831197    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818c8b69-f230-4bf7-b050-ae9e23c0281b-tmp-dir\") pod \"node-resolver-k7f2q\" (UID: \"818c8b69-f230-4bf7-b050-ae9e23c0281b\") " pod="openshift-dns/node-resolver-k7f2q"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831224    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-registration-dir\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831250    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-var-lib-kubelet\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831278    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-multus-conf-dir\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831306    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-cni-bin\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831335    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84c96591-3b96-4722-a376-ea2b0728b358-ovn-node-metrics-cert\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831363    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-lib-modules\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831391    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-var-lib-kubelet\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831419    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-tmp\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.832772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.831769    2551 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.832212    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-etc-openvswitch\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.832278    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-sysctl-d\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:59.832418    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:59.832507    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs podName:a0d57ecc-0561-4cce-855e-0281fb270d89 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:00.332471395 +0000 UTC m=+3.026518809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs") pod "network-metrics-daemon-r7fbk" (UID: "a0d57ecc-0561-4cce-855e-0281fb270d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.832768    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84c96591-3b96-4722-a376-ea2b0728b358-ovnkube-script-lib\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.832924    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84c96591-3b96-4722-a376-ea2b0728b358-ovnkube-config\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.832983    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4034a0ac-d7a3-4689-8f4b-2281d0d0722a-iptables-alerter-script\") pod \"iptables-alerter-jqkv8\" (UID: \"4034a0ac-d7a3-4689-8f4b-2281d0d0722a\") " pod="openshift-network-operator/iptables-alerter-jqkv8"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.832995    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/818c8b69-f230-4bf7-b050-ae9e23c0281b-tmp-dir\") pod \"node-resolver-k7f2q\" (UID: \"818c8b69-f230-4bf7-b050-ae9e23c0281b\") " pod="openshift-dns/node-resolver-k7f2q"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833065    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-registration-dir\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833118    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4034a0ac-d7a3-4689-8f4b-2281d0d0722a-host-slash\") pod \"iptables-alerter-jqkv8\" (UID: \"4034a0ac-d7a3-4689-8f4b-2281d0d0722a\") " pod="openshift-network-operator/iptables-alerter-jqkv8"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833126    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-kubelet\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833216    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833247    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-modprobe-d\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833271    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833298    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-kubernetes\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833341    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-run-ovn\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.833656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833365    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-socket-dir\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833405    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-host\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833527    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-run-netns\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833560    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be479e0b-db97-4752-9f8a-4676b12319c5-konnectivity-ca\") pod \"konnectivity-agent-5z6n7\" (UID: \"be479e0b-db97-4752-9f8a-4676b12319c5\") " pod="kube-system/konnectivity-agent-5z6n7"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833571    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-node-log\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833614    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-systemd\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833633    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-log-socket\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833674    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-slash\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833685    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-sys\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833716    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-run\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833729    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833797    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-sys-fs\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.833869    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b96b8478-7682-4a69-affb-0fc5239b5a4e-device-dir\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.834007    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-sysconfig\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.834524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.834455    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84c96591-3b96-4722-a376-ea2b0728b358-env-overrides\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.835569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.834608    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-lib-modules\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.835569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.834661    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-cni-bin\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.835569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.834732    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-var-lib-kubelet\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.835569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.834790    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-run-systemd\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.835569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.834830    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-var-lib-openvswitch\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.835569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.834876    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-run-ovn-kubernetes\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.835569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.835283    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-run-openvswitch\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.835569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.835314    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/818c8b69-f230-4bf7-b050-ae9e23c0281b-hosts-file\") pod \"node-resolver-k7f2q\" (UID: \"818c8b69-f230-4bf7-b050-ae9e23c0281b\") " pod="openshift-dns/node-resolver-k7f2q"
Jul 31 23:45:59.835569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.835319    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-systemd-units\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.835569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.835335    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84c96591-3b96-4722-a376-ea2b0728b358-host-cni-netd\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.841119 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.841098    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-sysctl-conf\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.845417 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.841383    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-etc-tuned\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.845417 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.841741    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84c96591-3b96-4722-a376-ea2b0728b358-ovn-node-metrics-cert\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.845417 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.842155    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-tmp\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.845417 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.842185    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be479e0b-db97-4752-9f8a-4676b12319c5-agent-certs\") pod \"konnectivity-agent-5z6n7\" (UID: \"be479e0b-db97-4752-9f8a-4676b12319c5\") " pod="kube-system/konnectivity-agent-5z6n7"
Jul 31 23:45:59.845838 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.845560    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn5b6\" (UniqueName: \"kubernetes.io/projected/6c720d73-4a3b-4b49-b8b5-e7ef71578c1c-kube-api-access-xn5b6\") pod \"tuned-9ppkm\" (UID: \"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c\") " pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:45:59.848483 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.848451    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hdfg\" (UniqueName: \"kubernetes.io/projected/a0d57ecc-0561-4cce-855e-0281fb270d89-kube-api-access-4hdfg\") pod \"network-metrics-daemon-r7fbk\" (UID: \"a0d57ecc-0561-4cce-855e-0281fb270d89\") " pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:45:59.849110 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.849046    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5tb2\" (UniqueName: \"kubernetes.io/projected/4034a0ac-d7a3-4689-8f4b-2281d0d0722a-kube-api-access-l5tb2\") pod \"iptables-alerter-jqkv8\" (UID: \"4034a0ac-d7a3-4689-8f4b-2281d0d0722a\") " pod="openshift-network-operator/iptables-alerter-jqkv8"
Jul 31 23:45:59.851487 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.851462    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz6jr\" (UniqueName: \"kubernetes.io/projected/84c96591-3b96-4722-a376-ea2b0728b358-kube-api-access-pz6jr\") pod \"ovnkube-node-b47km\" (UID: \"84c96591-3b96-4722-a376-ea2b0728b358\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:45:59.851921 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.851876    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99ctn\" (UniqueName: \"kubernetes.io/projected/818c8b69-f230-4bf7-b050-ae9e23c0281b-kube-api-access-99ctn\") pod \"node-resolver-k7f2q\" (UID: \"818c8b69-f230-4bf7-b050-ae9e23c0281b\") " pod="openshift-dns/node-resolver-k7f2q"
Jul 31 23:45:59.853479 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.853452    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvd9\" (UniqueName: \"kubernetes.io/projected/b96b8478-7682-4a69-affb-0fc5239b5a4e-kube-api-access-2pvd9\") pod \"aws-ebs-csi-driver-node-smhh9\" (UID: \"b96b8478-7682-4a69-affb-0fc5239b5a4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:45:59.932606 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932566    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61b45fe3-b6e6-47f9-8974-11cb808bc9f0-host\") pod \"node-ca-wkzlt\" (UID: \"61b45fe3-b6e6-47f9-8974-11cb808bc9f0\") " pod="openshift-image-registry/node-ca-wkzlt"
Jul 31 23:45:59.932785 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932624    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-os-release\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.932785 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932654    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-etc-kubernetes\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.932785 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932679    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/502fbd41-3e97-4d6e-ac72-d6a05311d675-cnibin\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.932785 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932704    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/502fbd41-3e97-4d6e-ac72-d6a05311d675-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.932785 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932735    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-run-netns\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.932785 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932760    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9mk\" (UniqueName: \"kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk\") pod \"network-check-target-j2rjc\" (UID: \"15c09ad2-cc88-49b1-b608-6f4bab832087\") " pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932800    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-run-multus-certs\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932827    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/502fbd41-3e97-4d6e-ac72-d6a05311d675-cni-binary-copy\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932853    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvcbm\" (UniqueName: \"kubernetes.io/projected/61b45fe3-b6e6-47f9-8974-11cb808bc9f0-kube-api-access-cvcbm\") pod \"node-ca-wkzlt\" (UID: \"61b45fe3-b6e6-47f9-8974-11cb808bc9f0\") " pod="openshift-image-registry/node-ca-wkzlt"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932877    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-var-lib-cni-bin\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932901    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-hostroot\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932925    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw75l\" (UniqueName: \"kubernetes.io/projected/502fbd41-3e97-4d6e-ac72-d6a05311d675-kube-api-access-cw75l\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.932955    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-cnibin\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933001    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-run-k8s-cni-cncf-io\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933026    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40e0e72a-edb8-4893-9b23-e12962884e30-cni-binary-copy\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933051    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vwb4\" (UniqueName: \"kubernetes.io/projected/40e0e72a-edb8-4893-9b23-e12962884e30-kube-api-access-8vwb4\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933077    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20d38ffc-045d-41cf-bcde-a74d3e0b8361-dbus\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933111    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-var-lib-cni-multus\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933130    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/502fbd41-3e97-4d6e-ac72-d6a05311d675-os-release\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.933150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933146    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20d38ffc-045d-41cf-bcde-a74d3e0b8361-kubelet-config\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933163    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-system-cni-dir\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933178    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-multus-cni-dir\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933193    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-multus-socket-dir-parent\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933208    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/502fbd41-3e97-4d6e-ac72-d6a05311d675-system-cni-dir\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933235    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/502fbd41-3e97-4d6e-ac72-d6a05311d675-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933251    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61b45fe3-b6e6-47f9-8974-11cb808bc9f0-serviceca\") pod \"node-ca-wkzlt\" (UID: \"61b45fe3-b6e6-47f9-8974-11cb808bc9f0\") " pod="openshift-image-registry/node-ca-wkzlt"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933274    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40e0e72a-edb8-4893-9b23-e12962884e30-multus-daemon-config\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933290    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933325    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/502fbd41-3e97-4d6e-ac72-d6a05311d675-tuning-conf-dir\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933356    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-var-lib-kubelet\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933377    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-multus-conf-dir\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933450    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-multus-conf-dir\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933494    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61b45fe3-b6e6-47f9-8974-11cb808bc9f0-host\") pod \"node-ca-wkzlt\" (UID: \"61b45fe3-b6e6-47f9-8974-11cb808bc9f0\") " pod="openshift-image-registry/node-ca-wkzlt"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933580    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-os-release\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933630    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-etc-kubernetes\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933674    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/502fbd41-3e97-4d6e-ac72-d6a05311d675-cnibin\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933809    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/502fbd41-3e97-4d6e-ac72-d6a05311d675-system-cni-dir\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.933861 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933838    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-run-netns\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.933949    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20d38ffc-045d-41cf-bcde-a74d3e0b8361-dbus\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934024    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-var-lib-cni-multus\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934087    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/502fbd41-3e97-4d6e-ac72-d6a05311d675-os-release\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934145    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-run-multus-certs\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934152    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-system-cni-dir\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934203    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-multus-cni-dir\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934262    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-multus-socket-dir-parent\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934299    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/502fbd41-3e97-4d6e-ac72-d6a05311d675-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934361    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-var-lib-kubelet\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934402    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40e0e72a-edb8-4893-9b23-e12962884e30-multus-daemon-config\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934415    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-cnibin\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934424    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/502fbd41-3e97-4d6e-ac72-d6a05311d675-tuning-conf-dir\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934482    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-hostroot\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:59.934511    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:59.934568    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret podName:20d38ffc-045d-41cf-bcde-a74d3e0b8361 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:00.434550831 +0000 UTC m=+3.128598244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret") pod "global-pull-secret-syncer-hfz7q" (UID: "20d38ffc-045d-41cf-bcde-a74d3e0b8361") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934654    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-var-lib-cni-bin\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.934775 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934775    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40e0e72a-edb8-4893-9b23-e12962884e30-host-run-k8s-cni-cncf-io\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.935613 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.934862    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20d38ffc-045d-41cf-bcde-a74d3e0b8361-kubelet-config\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:45:59.935613 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.935258    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61b45fe3-b6e6-47f9-8974-11cb808bc9f0-serviceca\") pod \"node-ca-wkzlt\" (UID: \"61b45fe3-b6e6-47f9-8974-11cb808bc9f0\") " pod="openshift-image-registry/node-ca-wkzlt"
Jul 31 23:45:59.935613 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.935307    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/502fbd41-3e97-4d6e-ac72-d6a05311d675-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.935613 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.935346    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40e0e72a-edb8-4893-9b23-e12962884e30-cni-binary-copy\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.935613 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.935498    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/502fbd41-3e97-4d6e-ac72-d6a05311d675-cni-binary-copy\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:45:59.945251 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:59.945193    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:45:59.945251 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:59.945230    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:45:59.945251 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:59.945246    2551 projected.go:196] Error preparing data for projected volume kube-api-access-2l9mk for pod openshift-network-diagnostics/network-check-target-j2rjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:59.945470 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:45:59.945307    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk podName:15c09ad2-cc88-49b1-b608-6f4bab832087 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:00.445289916 +0000 UTC m=+3.139337331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2l9mk" (UniqueName: "kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk") pod "network-check-target-j2rjc" (UID: "15c09ad2-cc88-49b1-b608-6f4bab832087") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:45:59.946992 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.946905    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" event={"ID":"d4efdb1fd75703bb7b34b57a3634782a","Type":"ContainerStarted","Data":"d488e3c866bfaf39ec5428550312ed6a9eab2153726960e8e37e8865243b0081"}
Jul 31 23:45:59.947933 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.947888    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvcbm\" (UniqueName: \"kubernetes.io/projected/61b45fe3-b6e6-47f9-8974-11cb808bc9f0-kube-api-access-cvcbm\") pod \"node-ca-wkzlt\" (UID: \"61b45fe3-b6e6-47f9-8974-11cb808bc9f0\") " pod="openshift-image-registry/node-ca-wkzlt"
Jul 31 23:45:59.947933 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.947915    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vwb4\" (UniqueName: \"kubernetes.io/projected/40e0e72a-edb8-4893-9b23-e12962884e30-kube-api-access-8vwb4\") pod \"multus-86226\" (UID: \"40e0e72a-edb8-4893-9b23-e12962884e30\") " pod="openshift-multus/multus-86226"
Jul 31 23:45:59.948594 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.948548    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" event={"ID":"41e595559883d998bfb59f4d787bbea5","Type":"ContainerStarted","Data":"fd964e0a63d45f6906e665a766bb3cb99538afa9de751afaad58646048debd39"}
Jul 31 23:45:59.949896 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:45:59.949877    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw75l\" (UniqueName: \"kubernetes.io/projected/502fbd41-3e97-4d6e-ac72-d6a05311d675-kube-api-access-cw75l\") pod \"multus-additional-cni-plugins-66n9v\" (UID: \"502fbd41-3e97-4d6e-ac72-d6a05311d675\") " pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:46:00.033896 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.033817    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jqkv8"
Jul 31 23:46:00.041388 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.041361    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:46:00.052466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.052431    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5z6n7"
Jul 31 23:46:00.057292 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.057268    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9"
Jul 31 23:46:00.066282 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.066259    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9ppkm"
Jul 31 23:46:00.074989 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.074957    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k7f2q"
Jul 31 23:46:00.085018 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.084996    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wkzlt"
Jul 31 23:46:00.093751 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.093732    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86226"
Jul 31 23:46:00.100545 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.100503    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-66n9v"
Jul 31 23:46:00.117696 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.117679    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:00.263559 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.263477    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:00.336931 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.336886    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs\") pod \"network-metrics-daemon-r7fbk\" (UID: \"a0d57ecc-0561-4cce-855e-0281fb270d89\") " pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:00.337090 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:00.337052    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:00.337131 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:00.337124    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs podName:a0d57ecc-0561-4cce-855e-0281fb270d89 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:01.337103594 +0000 UTC m=+4.031150994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs") pod "network-metrics-daemon-r7fbk" (UID: "a0d57ecc-0561-4cce-855e-0281fb270d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:00.437529 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.437489    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:00.437727 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:00.437619    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:00.437727 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:00.437696    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret podName:20d38ffc-045d-41cf-bcde-a74d3e0b8361 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:01.437677004 +0000 UTC m=+4.131724400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret") pod "global-pull-secret-syncer-hfz7q" (UID: "20d38ffc-045d-41cf-bcde-a74d3e0b8361") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:00.538100 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.538011    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9mk\" (UniqueName: \"kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk\") pod \"network-check-target-j2rjc\" (UID: \"15c09ad2-cc88-49b1-b608-6f4bab832087\") " pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:00.538253 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:00.538169    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:00.538253 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:00.538191    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:00.538253 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:00.538203    2551 projected.go:196] Error preparing data for projected volume kube-api-access-2l9mk for pod openshift-network-diagnostics/network-check-target-j2rjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:00.538411 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:00.538271    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk podName:15c09ad2-cc88-49b1-b608-6f4bab832087 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:01.538249852 +0000 UTC m=+4.232297270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l9mk" (UniqueName: "kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk") pod "network-check-target-j2rjc" (UID: "15c09ad2-cc88-49b1-b608-6f4bab832087") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:00.779433 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.779388    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:40:59 +0000 UTC" deadline="2028-04-14 01:48:17.72400411 +0000 UTC"
Jul 31 23:46:00.779433 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.779426    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14930h2m16.944582726s"
Jul 31 23:46:00.941413 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:00.941345    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:00.941545 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:00.941455    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:00.983818 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:00.983791    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b45fe3_b6e6_47f9_8974_11cb808bc9f0.slice/crio-02d5ebbb3bc2644cbe8ca13d7b2c92d17877fdc9e9f396e8426d16918b690937 WatchSource:0}: Error finding container 02d5ebbb3bc2644cbe8ca13d7b2c92d17877fdc9e9f396e8426d16918b690937: Status 404 returned error can't find the container with id 02d5ebbb3bc2644cbe8ca13d7b2c92d17877fdc9e9f396e8426d16918b690937
Jul 31 23:46:00.985372 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:00.985345    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe479e0b_db97_4752_9f8a_4676b12319c5.slice/crio-03ee6d9bfe893c6753730d06f7fd6566eaae0787a3542ca087ae79b9afdf0286 WatchSource:0}: Error finding container 03ee6d9bfe893c6753730d06f7fd6566eaae0787a3542ca087ae79b9afdf0286: Status 404 returned error can't find the container with id 03ee6d9bfe893c6753730d06f7fd6566eaae0787a3542ca087ae79b9afdf0286
Jul 31 23:46:00.987301 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:00.987193    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818c8b69_f230_4bf7_b050_ae9e23c0281b.slice/crio-0b78b7554e3c848f82c28713926bf48284615e7ab05e449e3ba8768149e01e93 WatchSource:0}: Error finding container 0b78b7554e3c848f82c28713926bf48284615e7ab05e449e3ba8768149e01e93: Status 404 returned error can't find the container with id 0b78b7554e3c848f82c28713926bf48284615e7ab05e449e3ba8768149e01e93
Jul 31 23:46:00.989205 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:00.989165    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e0e72a_edb8_4893_9b23_e12962884e30.slice/crio-4a71c59e1f2af966515ea2d3bf964286953b376d014e8d7ef09853a70f340028 WatchSource:0}: Error finding container 4a71c59e1f2af966515ea2d3bf964286953b376d014e8d7ef09853a70f340028: Status 404 returned error can't find the container with id 4a71c59e1f2af966515ea2d3bf964286953b376d014e8d7ef09853a70f340028
Jul 31 23:46:00.991357 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:00.991067    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502fbd41_3e97_4d6e_ac72_d6a05311d675.slice/crio-d86701ed27c18b88b8763fbabb69e94f52102b209c8cfbc582617e85a0b9d11a WatchSource:0}: Error finding container d86701ed27c18b88b8763fbabb69e94f52102b209c8cfbc582617e85a0b9d11a: Status 404 returned error can't find the container with id d86701ed27c18b88b8763fbabb69e94f52102b209c8cfbc582617e85a0b9d11a
Jul 31 23:46:00.991357 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:00.991248    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c96591_3b96_4722_a376_ea2b0728b358.slice/crio-63ab79b8b81f8289c40eb82a03570eb38fa22f37238503df36823bdf14a0fbae WatchSource:0}: Error finding container 63ab79b8b81f8289c40eb82a03570eb38fa22f37238503df36823bdf14a0fbae: Status 404 returned error can't find the container with id 63ab79b8b81f8289c40eb82a03570eb38fa22f37238503df36823bdf14a0fbae
Jul 31 23:46:00.994075 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:00.994015    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96b8478_7682_4a69_affb_0fc5239b5a4e.slice/crio-d9d2e48aa0bf3c52f275ad4e336e61de1ac4243f041d4c0a0e752386685ec477 WatchSource:0}: Error finding container d9d2e48aa0bf3c52f275ad4e336e61de1ac4243f041d4c0a0e752386685ec477: Status 404 returned error can't find the container with id d9d2e48aa0bf3c52f275ad4e336e61de1ac4243f041d4c0a0e752386685ec477
Jul 31 23:46:00.996362 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:00.995165    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c720d73_4a3b_4b49_b8b5_e7ef71578c1c.slice/crio-fd65232bd0066b2f86685cbdbab2a5c4f823b6fb1673a2a1b77f9ef86bd8945f WatchSource:0}: Error finding container fd65232bd0066b2f86685cbdbab2a5c4f823b6fb1673a2a1b77f9ef86bd8945f: Status 404 returned error can't find the container with id fd65232bd0066b2f86685cbdbab2a5c4f823b6fb1673a2a1b77f9ef86bd8945f
Jul 31 23:46:00.998046 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:00.997634    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4034a0ac_d7a3_4689_8f4b_2281d0d0722a.slice/crio-cf33db06b215008dc0ebcecf254af41fce1726d96a3969a2fcb73e0df8f6e76e WatchSource:0}: Error finding container cf33db06b215008dc0ebcecf254af41fce1726d96a3969a2fcb73e0df8f6e76e: Status 404 returned error can't find the container with id cf33db06b215008dc0ebcecf254af41fce1726d96a3969a2fcb73e0df8f6e76e
Jul 31 23:46:01.344055 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:01.344020    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs\") pod \"network-metrics-daemon-r7fbk\" (UID: \"a0d57ecc-0561-4cce-855e-0281fb270d89\") " pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:01.344185 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:01.344164    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:01.344247 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:01.344237    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs podName:a0d57ecc-0561-4cce-855e-0281fb270d89 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:03.34421945 +0000 UTC m=+6.038266855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs") pod "network-metrics-daemon-r7fbk" (UID: "a0d57ecc-0561-4cce-855e-0281fb270d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:01.445324 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:01.445272    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:01.445493 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:01.445452    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:01.445551 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:01.445511    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret podName:20d38ffc-045d-41cf-bcde-a74d3e0b8361 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:03.445497184 +0000 UTC m=+6.139544579 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret") pod "global-pull-secret-syncer-hfz7q" (UID: "20d38ffc-045d-41cf-bcde-a74d3e0b8361") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:01.546852 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:01.546222    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9mk\" (UniqueName: \"kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk\") pod \"network-check-target-j2rjc\" (UID: \"15c09ad2-cc88-49b1-b608-6f4bab832087\") " pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:01.546852 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:01.546381    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:01.546852 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:01.546402    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:01.546852 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:01.546415    2551 projected.go:196] Error preparing data for projected volume kube-api-access-2l9mk for pod openshift-network-diagnostics/network-check-target-j2rjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:01.546852 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:01.546470    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk podName:15c09ad2-cc88-49b1-b608-6f4bab832087 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:03.546453907 +0000 UTC m=+6.240501315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l9mk" (UniqueName: "kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk") pod "network-check-target-j2rjc" (UID: "15c09ad2-cc88-49b1-b608-6f4bab832087") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:01.779746 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:01.779681    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:40:59 +0000 UTC" deadline="2028-05-10 14:52:10.014998576 +0000 UTC"
Jul 31 23:46:01.779746 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:01.779744    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15567h6m8.235259401s"
Jul 31 23:46:01.943132 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:01.943051    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:01.943332 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:01.943207    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:01.943552 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:01.943055    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:01.943668 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:01.943633    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:01.971483 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:01.970576    2551 generic.go:356] "Generic (PLEG): container finished" podID="41e595559883d998bfb59f4d787bbea5" containerID="cbdba87db7b200b34405c433c83426527f9dc79e1cbdb44b58a6361c9956423b" exitCode=0
Jul 31 23:46:01.971483 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:01.970670    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" event={"ID":"41e595559883d998bfb59f4d787bbea5","Type":"ContainerDied","Data":"cbdba87db7b200b34405c433c83426527f9dc79e1cbdb44b58a6361c9956423b"}
Jul 31 23:46:01.976758 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:01.976683    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9ppkm" event={"ID":"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c","Type":"ContainerStarted","Data":"fd65232bd0066b2f86685cbdbab2a5c4f823b6fb1673a2a1b77f9ef86bd8945f"}
Jul 31 23:46:01.979432 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:01.979329    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9" event={"ID":"b96b8478-7682-4a69-affb-0fc5239b5a4e","Type":"ContainerStarted","Data":"d9d2e48aa0bf3c52f275ad4e336e61de1ac4243f041d4c0a0e752386685ec477"}
Jul 31 23:46:01.987522 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:01.987428    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47km" event={"ID":"84c96591-3b96-4722-a376-ea2b0728b358","Type":"ContainerStarted","Data":"63ab79b8b81f8289c40eb82a03570eb38fa22f37238503df36823bdf14a0fbae"}
Jul 31 23:46:02.006476 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:02.006404    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k7f2q" event={"ID":"818c8b69-f230-4bf7-b050-ae9e23c0281b","Type":"ContainerStarted","Data":"0b78b7554e3c848f82c28713926bf48284615e7ab05e449e3ba8768149e01e93"}
Jul 31 23:46:02.020550 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:02.020515    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wkzlt" event={"ID":"61b45fe3-b6e6-47f9-8974-11cb808bc9f0","Type":"ContainerStarted","Data":"02d5ebbb3bc2644cbe8ca13d7b2c92d17877fdc9e9f396e8426d16918b690937"}
Jul 31 23:46:02.028627 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:02.028593    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" event={"ID":"d4efdb1fd75703bb7b34b57a3634782a","Type":"ContainerStarted","Data":"2bb04c2e2295245aaa8c1061de1d4f99d4482115c4716f3e2d1005264e7807ac"}
Jul 31 23:46:02.038510 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:02.038477    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jqkv8" event={"ID":"4034a0ac-d7a3-4689-8f4b-2281d0d0722a","Type":"ContainerStarted","Data":"cf33db06b215008dc0ebcecf254af41fce1726d96a3969a2fcb73e0df8f6e76e"}
Jul 31 23:46:02.042448 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:02.042206    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66n9v" event={"ID":"502fbd41-3e97-4d6e-ac72-d6a05311d675","Type":"ContainerStarted","Data":"d86701ed27c18b88b8763fbabb69e94f52102b209c8cfbc582617e85a0b9d11a"}
Jul 31 23:46:02.060134 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:02.060033    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86226" event={"ID":"40e0e72a-edb8-4893-9b23-e12962884e30","Type":"ContainerStarted","Data":"4a71c59e1f2af966515ea2d3bf964286953b376d014e8d7ef09853a70f340028"}
Jul 31 23:46:02.067452 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:02.067415    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5z6n7" event={"ID":"be479e0b-db97-4752-9f8a-4676b12319c5","Type":"ContainerStarted","Data":"03ee6d9bfe893c6753730d06f7fd6566eaae0787a3542ca087ae79b9afdf0286"}
Jul 31 23:46:02.942194 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:02.942116    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:02.942656 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:02.942251    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:03.078062 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:03.077986    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" event={"ID":"41e595559883d998bfb59f4d787bbea5","Type":"ContainerStarted","Data":"6261794cfc06319699de3b1fa6027f1c4c5580d6186d2ac02cf14025784d26bb"}
Jul 31 23:46:03.095531 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:03.094949    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-136.ec2.internal" podStartSLOduration=5.094912839 podStartE2EDuration="5.094912839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:02.042411915 +0000 UTC m=+4.736459332" watchObservedRunningTime="2026-07-31 23:46:03.094912839 +0000 UTC m=+5.788960257"
Jul 31 23:46:03.362269 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:03.362152    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs\") pod \"network-metrics-daemon-r7fbk\" (UID: \"a0d57ecc-0561-4cce-855e-0281fb270d89\") " pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:03.362449 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:03.362349    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:03.362449 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:03.362410    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs podName:a0d57ecc-0561-4cce-855e-0281fb270d89 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:07.362391152 +0000 UTC m=+10.056438560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs") pod "network-metrics-daemon-r7fbk" (UID: "a0d57ecc-0561-4cce-855e-0281fb270d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:03.463380 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:03.463346    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:03.463567 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:03.463497    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:03.463567 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:03.463555    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret podName:20d38ffc-045d-41cf-bcde-a74d3e0b8361 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:07.463538634 +0000 UTC m=+10.157586036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret") pod "global-pull-secret-syncer-hfz7q" (UID: "20d38ffc-045d-41cf-bcde-a74d3e0b8361") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:03.564558 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:03.564248    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9mk\" (UniqueName: \"kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk\") pod \"network-check-target-j2rjc\" (UID: \"15c09ad2-cc88-49b1-b608-6f4bab832087\") " pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:03.564558 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:03.564417    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:03.564558 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:03.564441    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:03.564558 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:03.564453    2551 projected.go:196] Error preparing data for projected volume kube-api-access-2l9mk for pod openshift-network-diagnostics/network-check-target-j2rjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:03.564558 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:03.564515    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk podName:15c09ad2-cc88-49b1-b608-6f4bab832087 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:07.564496511 +0000 UTC m=+10.258543909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l9mk" (UniqueName: "kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk") pod "network-check-target-j2rjc" (UID: "15c09ad2-cc88-49b1-b608-6f4bab832087") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:03.950089 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:03.950055    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:03.950623 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:03.950092    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:03.950623 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:03.950249    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:03.950623 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:03.950361    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:04.941324 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:04.941288    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:04.941509 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:04.941413    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:05.941152 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:05.941120    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:05.941152 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:05.941147    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:05.941695 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:05.941260    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:05.941695 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:05.941554    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:06.941490 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:06.941318    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:06.941490 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:06.941458    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:07.395502 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:07.395414    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs\") pod \"network-metrics-daemon-r7fbk\" (UID: \"a0d57ecc-0561-4cce-855e-0281fb270d89\") " pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:07.395671 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:07.395556    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:07.395671 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:07.395641    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs podName:a0d57ecc-0561-4cce-855e-0281fb270d89 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:15.395617908 +0000 UTC m=+18.089665313 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs") pod "network-metrics-daemon-r7fbk" (UID: "a0d57ecc-0561-4cce-855e-0281fb270d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:07.495794 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:07.495753    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:07.495997 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:07.495949    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:07.496073 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:07.496012    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret podName:20d38ffc-045d-41cf-bcde-a74d3e0b8361 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:15.495995901 +0000 UTC m=+18.190043294 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret") pod "global-pull-secret-syncer-hfz7q" (UID: "20d38ffc-045d-41cf-bcde-a74d3e0b8361") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:07.596492 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:07.596429    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9mk\" (UniqueName: \"kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk\") pod \"network-check-target-j2rjc\" (UID: \"15c09ad2-cc88-49b1-b608-6f4bab832087\") " pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:07.596672 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:07.596570    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:07.596672 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:07.596592    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:07.596672 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:07.596606    2551 projected.go:196] Error preparing data for projected volume kube-api-access-2l9mk for pod openshift-network-diagnostics/network-check-target-j2rjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:07.596672 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:07.596662    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk podName:15c09ad2-cc88-49b1-b608-6f4bab832087 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:15.596643764 +0000 UTC m=+18.290691171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l9mk" (UniqueName: "kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk") pod "network-check-target-j2rjc" (UID: "15c09ad2-cc88-49b1-b608-6f4bab832087") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:07.943422 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:07.942767    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:07.943422 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:07.942891    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:07.943422 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:07.943284    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:07.943422 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:07.943375    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:08.941790 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:08.941753    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:08.942004 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:08.941892    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:09.941774 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:09.941745    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:09.942235 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:09.941877    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:09.942235 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:09.941751    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:09.942362 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:09.942302    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:10.941129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:10.941096    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:10.941290 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:10.941228    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:11.941255 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:11.941217    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:11.941702 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:11.941221    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:11.941702 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:11.941365    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:11.941702 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:11.941390    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:12.941673 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:12.941636    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:12.942088 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:12.941749    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:13.941984 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:13.941948    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:13.942399 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:13.941994    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:13.942399 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:13.942080    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:13.942399 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:13.942207    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:14.941860 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:14.941823    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:14.942131 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:14.941964    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:15.456208 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:15.456165    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs\") pod \"network-metrics-daemon-r7fbk\" (UID: \"a0d57ecc-0561-4cce-855e-0281fb270d89\") " pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:15.456359 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:15.456325    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:15.456426 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:15.456406    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs podName:a0d57ecc-0561-4cce-855e-0281fb270d89 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:31.456385058 +0000 UTC m=+34.150432475 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs") pod "network-metrics-daemon-r7fbk" (UID: "a0d57ecc-0561-4cce-855e-0281fb270d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:15.557198 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:15.557160    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:15.557363 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:15.557305    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:15.557403 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:15.557377    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret podName:20d38ffc-045d-41cf-bcde-a74d3e0b8361 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:31.557356884 +0000 UTC m=+34.251404330 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret") pod "global-pull-secret-syncer-hfz7q" (UID: "20d38ffc-045d-41cf-bcde-a74d3e0b8361") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:15.657797 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:15.657766    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9mk\" (UniqueName: \"kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk\") pod \"network-check-target-j2rjc\" (UID: \"15c09ad2-cc88-49b1-b608-6f4bab832087\") " pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:15.657952 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:15.657938    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:15.657999 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:15.657955    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:15.657999 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:15.657964    2551 projected.go:196] Error preparing data for projected volume kube-api-access-2l9mk for pod openshift-network-diagnostics/network-check-target-j2rjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:15.658058 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:15.658014    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk podName:15c09ad2-cc88-49b1-b608-6f4bab832087 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:31.657997091 +0000 UTC m=+34.352044500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l9mk" (UniqueName: "kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk") pod "network-check-target-j2rjc" (UID: "15c09ad2-cc88-49b1-b608-6f4bab832087") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:15.941674 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:15.941643    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:15.941855 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:15.941768    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:15.941941 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:15.941856    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:15.942015 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:15.941989    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:16.941609 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:16.941578    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:16.942051 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:16.941691    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:17.942535 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:17.942491    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:17.942991 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:17.942603    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:17.942991 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:17.942666    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:17.942991 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:17.942795    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:18.940949 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:18.940888    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:18.941148 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:18.941060    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:19.942588 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:19.942022    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:19.942588 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:19.942022    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:19.942588 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:19.942536    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:19.943357 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:19.942614    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:20.110780 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.110756    2551 generic.go:356] "Generic (PLEG): container finished" podID="502fbd41-3e97-4d6e-ac72-d6a05311d675" containerID="7e1d615bf3191f88d2cfd1f108765b15763ec0c2d18e058127a2f00541182ff0" exitCode=0
Jul 31 23:46:20.110870 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.110820    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66n9v" event={"ID":"502fbd41-3e97-4d6e-ac72-d6a05311d675","Type":"ContainerDied","Data":"7e1d615bf3191f88d2cfd1f108765b15763ec0c2d18e058127a2f00541182ff0"}
Jul 31 23:46:20.112215 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.112195    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86226" event={"ID":"40e0e72a-edb8-4893-9b23-e12962884e30","Type":"ContainerStarted","Data":"1026ec722287aef2e905e94d65ef17193aa2c4ec9e73ccfcf2ae36a22cae35bf"}
Jul 31 23:46:20.113668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.113601    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5z6n7" event={"ID":"be479e0b-db97-4752-9f8a-4676b12319c5","Type":"ContainerStarted","Data":"d66e48b9c720a4661a65812a1189c7ae3be5b3fffa5c92e63159b4d53c2ad245"}
Jul 31 23:46:20.115085 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.115062    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9ppkm" event={"ID":"6c720d73-4a3b-4b49-b8b5-e7ef71578c1c","Type":"ContainerStarted","Data":"43473aa27d3097f14f7317374e68480ac7406b625f5a75c73f7003ff53020237"}
Jul 31 23:46:20.116444 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.116426    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9" event={"ID":"b96b8478-7682-4a69-affb-0fc5239b5a4e","Type":"ContainerStarted","Data":"b2be034fd67ad2f241b8776cba73fe774a0c878c9df9de122bba1ec073a213cd"}
Jul 31 23:46:20.119699 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.119674    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47km" event={"ID":"84c96591-3b96-4722-a376-ea2b0728b358","Type":"ContainerStarted","Data":"bb1c33cb01f29e2b4b8866e9a19cf5f7f38a8265e1bde83ebb6a5a1d97e10a52"}
Jul 31 23:46:20.119797 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.119743    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47km" event={"ID":"84c96591-3b96-4722-a376-ea2b0728b358","Type":"ContainerStarted","Data":"34a259aae19db580f4536587cad0c297829aa6566cc3380e677982a5f5deabde"}
Jul 31 23:46:20.119797 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.119787    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47km" event={"ID":"84c96591-3b96-4722-a376-ea2b0728b358","Type":"ContainerStarted","Data":"c91b1d0821a18557ba2981b989feab035e6f41976dd924888d6789f36153e5d3"}
Jul 31 23:46:20.119900 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.119829    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47km" event={"ID":"84c96591-3b96-4722-a376-ea2b0728b358","Type":"ContainerStarted","Data":"7ee8cbaf16acfa2237be4109268c0a4984046d5856091963f15d7c8f23ddd2a4"}
Jul 31 23:46:20.119900 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.119873    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47km" event={"ID":"84c96591-3b96-4722-a376-ea2b0728b358","Type":"ContainerStarted","Data":"e341f67806a9ee59f6fcf34bbd26db831f3f7293536d27956c031d9c4329b1c2"}
Jul 31 23:46:20.121800 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.121778    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k7f2q" event={"ID":"818c8b69-f230-4bf7-b050-ae9e23c0281b","Type":"ContainerStarted","Data":"c55fe66e6af10bb0308e6a7c0015e322446dacdaa0cfcf0b7c863f324c21e743"}
Jul 31 23:46:20.123153 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.123133    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wkzlt" event={"ID":"61b45fe3-b6e6-47f9-8974-11cb808bc9f0","Type":"ContainerStarted","Data":"cf2f0705622d4c902d9f288de595591e2b3cb42c604fd1779e3ee29f90f09fad"}
Jul 31 23:46:20.130728 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.130694    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-136.ec2.internal" podStartSLOduration=22.130682293 podStartE2EDuration="22.130682293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:03.095504128 +0000 UTC m=+5.789551546" watchObservedRunningTime="2026-07-31 23:46:20.130682293 +0000 UTC m=+22.824729708"
Jul 31 23:46:20.145987 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.145953    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9ppkm" podStartSLOduration=3.522918194 podStartE2EDuration="22.145945736s" totalImagesPullingTime="18.623027542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:20.14576812 +0000 UTC m=+22.839815546" watchObservedRunningTime="2026-07-31 23:46:20.145945736 +0000 UTC m=+22.839993153"
Jul 31 23:46:20.161800 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.161758    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-k7f2q" podStartSLOduration=3.551332264 podStartE2EDuration="22.1617431s" totalImagesPullingTime="18.610410836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:20.161235444 +0000 UTC m=+22.855282862" watchObservedRunningTime="2026-07-31 23:46:20.1617431 +0000 UTC m=+22.855790516"
Jul 31 23:46:20.187940 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.187887    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5z6n7" podStartSLOduration=11.59795372 podStartE2EDuration="22.187872325s" totalImagesPullingTime="10.589918605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:20.173304791 +0000 UTC m=+22.867352208" watchObservedRunningTime="2026-07-31 23:46:20.187872325 +0000 UTC m=+22.881919733"
Jul 31 23:46:20.188294 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.188267    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wkzlt" podStartSLOduration=3.5753635729999997 podStartE2EDuration="22.188258351s" totalImagesPullingTime="18.612894778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:20.188088643 +0000 UTC m=+22.882136058" watchObservedRunningTime="2026-07-31 23:46:20.188258351 +0000 UTC m=+22.882305767"
Jul 31 23:46:20.205119 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.205081    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-86226" podStartSLOduration=3.539046968 podStartE2EDuration="22.205068489s" totalImagesPullingTime="18.666021521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:20.204795336 +0000 UTC m=+22.898842753" watchObservedRunningTime="2026-07-31 23:46:20.205068489 +0000 UTC m=+22.899115905"
Jul 31 23:46:20.347498 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.347400    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:20.429227 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.429192    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:20.530876 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.530835    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:20.942036 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:20.942008    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:20.942221 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:20.942138    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:21.128391 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.128350    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47km" event={"ID":"84c96591-3b96-4722-a376-ea2b0728b358","Type":"ContainerStarted","Data":"7ae7acbb61f59f9156263beef815254535906e83542836bf82dc14807c8413f8"}
Jul 31 23:46:21.129977 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.129950    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jqkv8" event={"ID":"4034a0ac-d7a3-4689-8f4b-2281d0d0722a","Type":"ContainerStarted","Data":"ff90bcb77b2f51542f87ce7f2ea3841f2eb9a125cc1efb2b454c75f883759f11"}
Jul 31 23:46:21.144258 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.144170    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jqkv8" podStartSLOduration=4.54683193 podStartE2EDuration="23.144151726s" totalImagesPullingTime="18.597319796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:21.143443159 +0000 UTC m=+23.837490574" watchObservedRunningTime="2026-07-31 23:46:21.144151726 +0000 UTC m=+23.838199143"
Jul 31 23:46:21.237487 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.237461    2551 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 23:46:21.239165 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.239138    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:21.296449 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.296416    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:21.386177 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.386144    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:21.688898 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.688868    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:21.781487 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.781366    2551 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T23:46:21.237483902Z","UUID":"26b58ab8-8421-44f5-b091-576259b00d83","Handler":null,"Name":"","Endpoint":""}
Jul 31 23:46:21.782773 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.782755    2551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 23:46:21.782887 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.782793    2551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 23:46:21.789892 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.789868    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:21.941465 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.941386    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:21.941465 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:21.941430    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:21.941685 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:21.941547    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:21.941685 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:21.941616    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:22.133598 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:22.133561    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9" event={"ID":"b96b8478-7682-4a69-affb-0fc5239b5a4e","Type":"ContainerStarted","Data":"ddfee6dafac4ce8833bbf2efe190c73744b7fee5f7fc6c366f210b01e26bb5b1"}
Jul 31 23:46:22.941148 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:22.940972    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:22.941329 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:22.941233    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:23.137687 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:23.137649    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9" event={"ID":"b96b8478-7682-4a69-affb-0fc5239b5a4e","Type":"ContainerStarted","Data":"6816e3a267756cf7095eaeec7e0008bc007d3f5083b6554569856ac204aa0905"}
Jul 31 23:46:23.142150 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:23.142055    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47km" event={"ID":"84c96591-3b96-4722-a376-ea2b0728b358","Type":"ContainerStarted","Data":"a14a489a9d87c488c2e43a80a6270144c4fb1b297955916241211c84e53123a2"}
Jul 31 23:46:23.157820 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:23.157774    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smhh9" podStartSLOduration=4.304955103 podStartE2EDuration="25.15776208s" totalImagesPullingTime="20.852806977s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:23.157306043 +0000 UTC m=+25.851353459" watchObservedRunningTime="2026-07-31 23:46:23.15776208 +0000 UTC m=+25.851809513"
Jul 31 23:46:23.362183 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:23.362148    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:23.492375 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:23.492329    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:23.590352 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:23.590317    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:23.804899 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:23.804864    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5z6n7"
Jul 31 23:46:23.805648 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:23.805618    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5z6n7"
Jul 31 23:46:23.941154 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:23.941122    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:23.941322 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:23.941240    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:23.941322 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:23.941280    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:23.941418 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:23.941365    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:24.144971 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:24.144873    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5z6n7"
Jul 31 23:46:24.145510 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:24.145430    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5z6n7"
Jul 31 23:46:24.203730 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:24.203696    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:24.941645 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:24.941409    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:24.941786 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:24.941679    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:25.148071 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:25.148038    2551 generic.go:356] "Generic (PLEG): container finished" podID="502fbd41-3e97-4d6e-ac72-d6a05311d675" containerID="dd578788391ad85d8d3227e7c81fe2ff735c885c7aef1ca3089d6a7712dd694b" exitCode=0
Jul 31 23:46:25.148825 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:25.148125    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66n9v" event={"ID":"502fbd41-3e97-4d6e-ac72-d6a05311d675","Type":"ContainerDied","Data":"dd578788391ad85d8d3227e7c81fe2ff735c885c7aef1ca3089d6a7712dd694b"}
Jul 31 23:46:25.152047 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:25.152026    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47km" event={"ID":"84c96591-3b96-4722-a376-ea2b0728b358","Type":"ContainerStarted","Data":"1e60c81fc86ef4542adc9414ff25fde48640d8175befa13ed8ff0e27f833aaf9"}
Jul 31 23:46:25.191086 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:25.191039    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b47km" podStartSLOduration=8.517665837 podStartE2EDuration="27.191023158s" totalImagesPullingTime="18.673357321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:25.189989032 +0000 UTC m=+27.884036441" watchObservedRunningTime="2026-07-31 23:46:25.191023158 +0000 UTC m=+27.885070567"
Jul 31 23:46:25.377902 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:25.377866    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:25.451800 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:25.451767    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:25.503934 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:25.503891    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:25.942020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:25.941975    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:25.942020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:25.941998    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:25.942277 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:25.942102    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:25.942277 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:25.942213    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:26.154669 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.154639    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:46:26.155158 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.154702    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:46:26.155158 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.154733    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:46:26.171101 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.171068    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:46:26.171276 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.171258    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:46:26.285360 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.285337    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:26.387972 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.387949    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:26.446522 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.446494    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hfz7q"]
Jul 31 23:46:26.446691 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.446655    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:26.446780 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:26.446760    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:26.447165 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.447143    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j2rjc"]
Jul 31 23:46:26.447242 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.447216    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:26.447303 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:26.447286    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:26.448851 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.448832    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r7fbk"]
Jul 31 23:46:26.448966 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.448903    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:26.449020 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:26.449004    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:26.773066 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.773037    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:26.832605 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:26.832574    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:27.157713 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:27.157631    2551 generic.go:356] "Generic (PLEG): container finished" podID="502fbd41-3e97-4d6e-ac72-d6a05311d675" containerID="7a32dc76bfead457a7dd338a59f4b2770051b3ecb9ee4fb857160cb6325e63c7" exitCode=0
Jul 31 23:46:27.158086 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:27.157726    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66n9v" event={"ID":"502fbd41-3e97-4d6e-ac72-d6a05311d675","Type":"ContainerDied","Data":"7a32dc76bfead457a7dd338a59f4b2770051b3ecb9ee4fb857160cb6325e63c7"}
Jul 31 23:46:27.179430 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:27.179388    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:27.274729 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:27.274696    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:27.918219 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:27.918195    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k7f2q_818c8b69-f230-4bf7-b050-ae9e23c0281b/dns-node-resolver/0.log"
Jul 31 23:46:27.942716 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:27.942688    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:27.942868 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:27.942722    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:27.942868 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:27.942794    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:27.942868 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:27.942811    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:27.943028 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:27.942880    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:27.943028 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:27.942966    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:28.212875 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:28.212802    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:28.313524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:28.313483    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:28.625625 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:28.625593    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:28.725809 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:28.725779    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:28.917260 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:28.917191    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:28.917754 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:28.917733    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wkzlt_61b45fe3-b6e6-47f9-8974-11cb808bc9f0/node-ca/0.log"
Jul 31 23:46:29.010444 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:29.010412    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:29.163756 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:29.163729    2551 generic.go:356] "Generic (PLEG): container finished" podID="502fbd41-3e97-4d6e-ac72-d6a05311d675" containerID="5154d4a7648e30ccbe49ae6c352c86c4ed043da49ddc5027762f346c898f43e3" exitCode=0
Jul 31 23:46:29.163895 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:29.163762    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66n9v" event={"ID":"502fbd41-3e97-4d6e-ac72-d6a05311d675","Type":"ContainerDied","Data":"5154d4a7648e30ccbe49ae6c352c86c4ed043da49ddc5027762f346c898f43e3"}
Jul 31 23:46:29.831878 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:29.831847    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:29.933360 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:29.933328    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:29.941361 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:29.941336    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:29.941361 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:29.941353    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:29.941535 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:29.941469    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:29.941595 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:29.941536    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:29.941595 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:29.941560    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:29.941664 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:29.941645    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:30.546243 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:30.546209    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:31.480581 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:31.480550    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs\") pod \"network-metrics-daemon-r7fbk\" (UID: \"a0d57ecc-0561-4cce-855e-0281fb270d89\") " pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:31.481067 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:31.480710    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:31.481067 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:31.480779    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs podName:a0d57ecc-0561-4cce-855e-0281fb270d89 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:03.480757718 +0000 UTC m=+66.174805111 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs") pod "network-metrics-daemon-r7fbk" (UID: "a0d57ecc-0561-4cce-855e-0281fb270d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:31.481435 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:31.481414    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:31.580878 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:31.580845    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:31.581057 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:31.581000    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:31.581057 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:31.581055    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret podName:20d38ffc-045d-41cf-bcde-a74d3e0b8361 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:03.581040184 +0000 UTC m=+66.275087578 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret") pod "global-pull-secret-syncer-hfz7q" (UID: "20d38ffc-045d-41cf-bcde-a74d3e0b8361") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:31.682102 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:31.682066    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9mk\" (UniqueName: \"kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk\") pod \"network-check-target-j2rjc\" (UID: \"15c09ad2-cc88-49b1-b608-6f4bab832087\") " pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:31.682287 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:31.682198    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:31.682287 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:31.682218    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:31.682287 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:31.682230    2551 projected.go:196] Error preparing data for projected volume kube-api-access-2l9mk for pod openshift-network-diagnostics/network-check-target-j2rjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:31.682287 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:31.682281    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk podName:15c09ad2-cc88-49b1-b608-6f4bab832087 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:03.682267559 +0000 UTC m=+66.376314966 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l9mk" (UniqueName: "kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk") pod "network-check-target-j2rjc" (UID: "15c09ad2-cc88-49b1-b608-6f4bab832087") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:31.941281 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:31.941198    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:31.941440 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:31.941198    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:31.941440 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:31.941344    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r7fbk" podUID="a0d57ecc-0561-4cce-855e-0281fb270d89"
Jul 31 23:46:31.941440 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:31.941423    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j2rjc" podUID="15c09ad2-cc88-49b1-b608-6f4bab832087"
Jul 31 23:46:31.941647 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:31.941202    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:31.941647 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:31.941548    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hfz7q" podUID="20d38ffc-045d-41cf-bcde-a74d3e0b8361"
Jul 31 23:46:32.451008 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:32.450784    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:32.587884 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:32.587856    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-136.ec2.internal" event="NodeReady"
Jul 31 23:46:32.588329 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:32.588038    2551 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 23:46:32.642833 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:32.642796    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:32.744055 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:32.744016    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:32.855532 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:32.855498    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:32.955445 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:32.955399    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:33.064533 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:33.064451    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:33.241466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:33.241430    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:33.932785 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:33.932668    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:33.941697 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:33.941678    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:46:33.941776 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:33.941681    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:46:33.941809 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:33.941802    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:46:33.944186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:33.944163    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c2rbb\""
Jul 31 23:46:33.944186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:33.944182    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 23:46:33.944408 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:33.944185    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:46:33.944802 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:33.944774    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 23:46:33.944955 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:33.944941    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:46:33.945048 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:33.945005    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45dqd\""
Jul 31 23:46:33.992505 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:33.992472    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:34.138664 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:34.138632    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:34.178084 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:34.178053    2551 generic.go:356] "Generic (PLEG): container finished" podID="502fbd41-3e97-4d6e-ac72-d6a05311d675" containerID="3461010252a4f4db9e0acd9b0dd1397fb4f25bd57f53be89ff1e93a1a66d8af3" exitCode=0
Jul 31 23:46:34.178084 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:34.178089    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66n9v" event={"ID":"502fbd41-3e97-4d6e-ac72-d6a05311d675","Type":"ContainerDied","Data":"3461010252a4f4db9e0acd9b0dd1397fb4f25bd57f53be89ff1e93a1a66d8af3"}
Jul 31 23:46:34.749101 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:34.749069    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:35.183005 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:35.182904    2551 generic.go:356] "Generic (PLEG): container finished" podID="502fbd41-3e97-4d6e-ac72-d6a05311d675" containerID="a6a103dcc6b98059be0d85629226136c88ff0d13b77eb108e18b96de99b2f884" exitCode=0
Jul 31 23:46:35.183005 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:35.182972    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66n9v" event={"ID":"502fbd41-3e97-4d6e-ac72-d6a05311d675","Type":"ContainerDied","Data":"a6a103dcc6b98059be0d85629226136c88ff0d13b77eb108e18b96de99b2f884"}
Jul 31 23:46:35.337035 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:35.336997    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:35.537199 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:35.537167    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:35.732454 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:35.732422    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:35.932619 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:35.932529    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:36.130052 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:36.130025    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:36.188483 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:36.188407    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66n9v" event={"ID":"502fbd41-3e97-4d6e-ac72-d6a05311d675","Type":"ContainerStarted","Data":"e6f9ad2fb3b6cf35e30628772cace1fd79864d89205b842b4e43ed082badc360"}
Jul 31 23:46:36.215699 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:36.215657    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-66n9v" podStartSLOduration=8.603461478 podStartE2EDuration="38.215644366s" totalImagesPullingTime="29.612182888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:36.214402408 +0000 UTC m=+38.908449824" watchObservedRunningTime="2026-07-31 23:46:36.215644366 +0000 UTC m=+38.909691779"
Jul 31 23:46:36.785266 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:36.785219    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:36.918060 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:36.918017    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:36.976504 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:36.976478    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:37.146992 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:37.146906    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:37.539466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:37.539420    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:37.741089 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:37.741057    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:37.801961 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:37.801870    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:37.950888 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:37.950856    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:38.926538 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:38.926491    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:39.027351 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:39.027321    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:39.143884 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:39.143809    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:39.335105 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:39.335075    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:39.537057 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:39.537025    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:40.741560 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:40.741523    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:40.949742 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:40.949698    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:41.142104 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:41.142021    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:41.288092 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:41.288063    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:41.388950 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:41.388879    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:41.540646 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:41.540614    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:42.538298 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:42.538261    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:42.736590 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:42.736556    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:42.935046 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:42.934977    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:43.135045 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:43.135012    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:44.835079 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:44.835046    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:44.896765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:44.896730    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:51.468014 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:51.467977    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:51.568462 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:51.568435    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:51.636464 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:51.636436    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:51.765117 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:51.765077    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:52.134129 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.134054    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:52.220527 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.220497    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:52.673184 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.673151    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:52.793952 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.793905    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77b58c48d7-pdhnl"]
Jul 31 23:46:52.795863 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.795848    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:52.799391 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.799368    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 31 23:46:52.799559 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.799540    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 31 23:46:52.799646 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.799615    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 31 23:46:52.799709 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.799687    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mwlw9\""
Jul 31 23:46:52.805628 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.805609    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 31 23:46:52.810216 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.810193    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bsrg5"]
Jul 31 23:46:52.812018 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.812002    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h9q9h"]
Jul 31 23:46:52.812132 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.812118    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:52.813832 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.813813    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:52.815972 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.815954    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77b58c48d7-pdhnl"]
Jul 31 23:46:52.816956 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.816935    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 23:46:52.817235 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.817182    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 23:46:52.817320 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.817244    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tss5j\""
Jul 31 23:46:52.817388 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.817313    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 23:46:52.817388 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.817360    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mw9xk\""
Jul 31 23:46:52.817488 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.817455    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 23:46:52.817688 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.817674    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 23:46:52.817969 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.817955    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 23:46:52.838809 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.838787    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bsrg5"]
Jul 31 23:46:52.845330 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.845309    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h9q9h"]
Jul 31 23:46:52.845644 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.845626    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:52.914263 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.914231    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jxsn8"]
Jul 31 23:46:52.916346 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.916321    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jxsn8"
Jul 31 23:46:52.924019 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.923968    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03207f34-03d6-4655-ba13-0de638a39a5d-registry-tls\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:52.924019 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.923994    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkr5d\" (UniqueName: \"kubernetes.io/projected/03207f34-03d6-4655-ba13-0de638a39a5d-kube-api-access-bkr5d\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:52.924019 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924014    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfvrh\" (UniqueName: \"kubernetes.io/projected/97fb1913-5741-455f-8595-177262ce365a-kube-api-access-lfvrh\") pod \"dns-default-bsrg5\" (UID: \"97fb1913-5741-455f-8595-177262ce365a\") " pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:52.924240 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924081    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03207f34-03d6-4655-ba13-0de638a39a5d-installation-pull-secrets\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:52.924240 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924121    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fb1913-5741-455f-8595-177262ce365a-tmp-dir\") pod \"dns-default-bsrg5\" (UID: \"97fb1913-5741-455f-8595-177262ce365a\") " pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:52.924240 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924153    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03207f34-03d6-4655-ba13-0de638a39a5d-ca-trust-extracted\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:52.924240 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924180    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-799bh\" (UniqueName: \"kubernetes.io/projected/f668c162-e18e-43ab-ad64-d2d290bf1301-kube-api-access-799bh\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:52.924240 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924210    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03207f34-03d6-4655-ba13-0de638a39a5d-trusted-ca\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:52.924240 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924234    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f668c162-e18e-43ab-ad64-d2d290bf1301-crio-socket\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:52.924526 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924304    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03207f34-03d6-4655-ba13-0de638a39a5d-registry-certificates\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:52.924526 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924328    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03207f34-03d6-4655-ba13-0de638a39a5d-bound-sa-token\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:52.924526 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924368    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97fb1913-5741-455f-8595-177262ce365a-metrics-tls\") pod \"dns-default-bsrg5\" (UID: \"97fb1913-5741-455f-8595-177262ce365a\") " pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:52.924526 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924386    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f668c162-e18e-43ab-ad64-d2d290bf1301-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:52.924526 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924444    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03207f34-03d6-4655-ba13-0de638a39a5d-image-registry-private-configuration\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:52.924526 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924516    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97fb1913-5741-455f-8595-177262ce365a-config-volume\") pod \"dns-default-bsrg5\" (UID: \"97fb1913-5741-455f-8595-177262ce365a\") " pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:52.924781 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924546    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f668c162-e18e-43ab-ad64-d2d290bf1301-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:52.924781 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.924584    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f668c162-e18e-43ab-ad64-d2d290bf1301-data-volume\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:52.925087 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.925072    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 23:46:52.925576 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.925561    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 23:46:52.926124 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.926111    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 23:46:52.927421 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.927407    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qx4sg\""
Jul 31 23:46:52.933907 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:52.933885    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jxsn8"]
Jul 31 23:46:53.018737 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.018708    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:53.025797 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.025773    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ab1356e-7c27-49ae-a3ad-700b2c0ac399-cert\") pod \"ingress-canary-jxsn8\" (UID: \"9ab1356e-7c27-49ae-a3ad-700b2c0ac399\") " pod="openshift-ingress-canary/ingress-canary-jxsn8"
Jul 31 23:46:53.025896 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.025811    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03207f34-03d6-4655-ba13-0de638a39a5d-registry-certificates\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.025896 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.025837    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03207f34-03d6-4655-ba13-0de638a39a5d-bound-sa-token\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.025896 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.025881    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97fb1913-5741-455f-8595-177262ce365a-metrics-tls\") pod \"dns-default-bsrg5\" (UID: \"97fb1913-5741-455f-8595-177262ce365a\") " pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:53.026084 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.025906    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f668c162-e18e-43ab-ad64-d2d290bf1301-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:53.026084 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.025957    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03207f34-03d6-4655-ba13-0de638a39a5d-image-registry-private-configuration\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.026084 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026004    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97fb1913-5741-455f-8595-177262ce365a-config-volume\") pod \"dns-default-bsrg5\" (UID: \"97fb1913-5741-455f-8595-177262ce365a\") " pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:53.026084 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026035    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f668c162-e18e-43ab-ad64-d2d290bf1301-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:53.026084 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026073    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f668c162-e18e-43ab-ad64-d2d290bf1301-data-volume\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:53.026322 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026106    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03207f34-03d6-4655-ba13-0de638a39a5d-registry-tls\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.026322 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026130    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkr5d\" (UniqueName: \"kubernetes.io/projected/03207f34-03d6-4655-ba13-0de638a39a5d-kube-api-access-bkr5d\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.026322 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026156    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqc5\" (UniqueName: \"kubernetes.io/projected/9ab1356e-7c27-49ae-a3ad-700b2c0ac399-kube-api-access-pxqc5\") pod \"ingress-canary-jxsn8\" (UID: \"9ab1356e-7c27-49ae-a3ad-700b2c0ac399\") " pod="openshift-ingress-canary/ingress-canary-jxsn8"
Jul 31 23:46:53.026322 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026185    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfvrh\" (UniqueName: \"kubernetes.io/projected/97fb1913-5741-455f-8595-177262ce365a-kube-api-access-lfvrh\") pod \"dns-default-bsrg5\" (UID: \"97fb1913-5741-455f-8595-177262ce365a\") " pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:53.026322 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026232    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03207f34-03d6-4655-ba13-0de638a39a5d-installation-pull-secrets\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.026322 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026254    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fb1913-5741-455f-8595-177262ce365a-tmp-dir\") pod \"dns-default-bsrg5\" (UID: \"97fb1913-5741-455f-8595-177262ce365a\") " pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:53.026322 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026281    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03207f34-03d6-4655-ba13-0de638a39a5d-ca-trust-extracted\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.026322 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026305    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-799bh\" (UniqueName: \"kubernetes.io/projected/f668c162-e18e-43ab-ad64-d2d290bf1301-kube-api-access-799bh\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:53.026687 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026336    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03207f34-03d6-4655-ba13-0de638a39a5d-trusted-ca\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.026687 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026361    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f668c162-e18e-43ab-ad64-d2d290bf1301-crio-socket\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:53.026687 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026447    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f668c162-e18e-43ab-ad64-d2d290bf1301-crio-socket\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:53.026687 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026598    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f668c162-e18e-43ab-ad64-d2d290bf1301-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:53.026884 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026714    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f668c162-e18e-43ab-ad64-d2d290bf1301-data-volume\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:53.026884 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026716    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03207f34-03d6-4655-ba13-0de638a39a5d-registry-certificates\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.026884 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026737    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97fb1913-5741-455f-8595-177262ce365a-config-volume\") pod \"dns-default-bsrg5\" (UID: \"97fb1913-5741-455f-8595-177262ce365a\") " pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:53.027061 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026978    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fb1913-5741-455f-8595-177262ce365a-tmp-dir\") pod \"dns-default-bsrg5\" (UID: \"97fb1913-5741-455f-8595-177262ce365a\") " pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:53.027061 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.026979    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03207f34-03d6-4655-ba13-0de638a39a5d-ca-trust-extracted\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.028058 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.028031    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03207f34-03d6-4655-ba13-0de638a39a5d-trusted-ca\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.030283 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.030255    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f668c162-e18e-43ab-ad64-d2d290bf1301-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:53.030283 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.030264    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97fb1913-5741-455f-8595-177262ce365a-metrics-tls\") pod \"dns-default-bsrg5\" (UID: \"97fb1913-5741-455f-8595-177262ce365a\") " pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:53.030283 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.030261    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03207f34-03d6-4655-ba13-0de638a39a5d-installation-pull-secrets\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.030443 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.030361    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03207f34-03d6-4655-ba13-0de638a39a5d-registry-tls\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.030443 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.030407    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03207f34-03d6-4655-ba13-0de638a39a5d-image-registry-private-configuration\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.037950 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.037896    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03207f34-03d6-4655-ba13-0de638a39a5d-bound-sa-token\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.038260 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.038239    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-799bh\" (UniqueName: \"kubernetes.io/projected/f668c162-e18e-43ab-ad64-d2d290bf1301-kube-api-access-799bh\") pod \"insights-runtime-extractor-h9q9h\" (UID: \"f668c162-e18e-43ab-ad64-d2d290bf1301\") " pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:53.039614 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.039593    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfvrh\" (UniqueName: \"kubernetes.io/projected/97fb1913-5741-455f-8595-177262ce365a-kube-api-access-lfvrh\") pod \"dns-default-bsrg5\" (UID: \"97fb1913-5741-455f-8595-177262ce365a\") " pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:53.042870 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.042852    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkr5d\" (UniqueName: \"kubernetes.io/projected/03207f34-03d6-4655-ba13-0de638a39a5d-kube-api-access-bkr5d\") pod \"image-registry-77b58c48d7-pdhnl\" (UID: \"03207f34-03d6-4655-ba13-0de638a39a5d\") " pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.105504 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.105474    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:53.121831 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.121807    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:53.126777 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.126752    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h9q9h"
Jul 31 23:46:53.126912 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.126895    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ab1356e-7c27-49ae-a3ad-700b2c0ac399-cert\") pod \"ingress-canary-jxsn8\" (UID: \"9ab1356e-7c27-49ae-a3ad-700b2c0ac399\") " pod="openshift-ingress-canary/ingress-canary-jxsn8"
Jul 31 23:46:53.127014 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.126999    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqc5\" (UniqueName: \"kubernetes.io/projected/9ab1356e-7c27-49ae-a3ad-700b2c0ac399-kube-api-access-pxqc5\") pod \"ingress-canary-jxsn8\" (UID: \"9ab1356e-7c27-49ae-a3ad-700b2c0ac399\") " pod="openshift-ingress-canary/ingress-canary-jxsn8"
Jul 31 23:46:53.129570 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.129547    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ab1356e-7c27-49ae-a3ad-700b2c0ac399-cert\") pod \"ingress-canary-jxsn8\" (UID: \"9ab1356e-7c27-49ae-a3ad-700b2c0ac399\") " pod="openshift-ingress-canary/ingress-canary-jxsn8"
Jul 31 23:46:53.141935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.141899    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqc5\" (UniqueName: \"kubernetes.io/projected/9ab1356e-7c27-49ae-a3ad-700b2c0ac399-kube-api-access-pxqc5\") pod \"ingress-canary-jxsn8\" (UID: \"9ab1356e-7c27-49ae-a3ad-700b2c0ac399\") " pod="openshift-ingress-canary/ingress-canary-jxsn8"
Jul 31 23:46:53.151128 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.151052    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:53.224549 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.224524    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jxsn8"
Jul 31 23:46:53.290787 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.290754    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77b58c48d7-pdhnl"]
Jul 31 23:46:53.295056 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:53.295027    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03207f34_03d6_4655_ba13_0de638a39a5d.slice/crio-97788f918203ede0e4eb0aa97e44c9202c1bf68104a06e9219e308a0cdfac91a WatchSource:0}: Error finding container 97788f918203ede0e4eb0aa97e44c9202c1bf68104a06e9219e308a0cdfac91a: Status 404 returned error can't find the container with id 97788f918203ede0e4eb0aa97e44c9202c1bf68104a06e9219e308a0cdfac91a
Jul 31 23:46:53.296784 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.296758    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bsrg5"]
Jul 31 23:46:53.297365 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:53.297338    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fb1913_5741_455f_8595_177262ce365a.slice/crio-b8952ad861a41826426ab2689e81dc7362226723bac722c12a284e0c92ecbc39 WatchSource:0}: Error finding container b8952ad861a41826426ab2689e81dc7362226723bac722c12a284e0c92ecbc39: Status 404 returned error can't find the container with id b8952ad861a41826426ab2689e81dc7362226723bac722c12a284e0c92ecbc39
Jul 31 23:46:53.344065 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.344023    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:53.398279 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.398254    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jxsn8"]
Jul 31 23:46:53.401376 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:53.401337    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab1356e_7c27_49ae_a3ad_700b2c0ac399.slice/crio-2fb209953bcccabd016728b627e993aed7b22aec1a50aa298c40d14fc70c9869 WatchSource:0}: Error finding container 2fb209953bcccabd016728b627e993aed7b22aec1a50aa298c40d14fc70c9869: Status 404 returned error can't find the container with id 2fb209953bcccabd016728b627e993aed7b22aec1a50aa298c40d14fc70c9869
Jul 31 23:46:53.402458 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.402436    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4"]
Jul 31 23:46:53.404227 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.404207    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4"
Jul 31 23:46:53.406256 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.406239    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 31 23:46:53.406345 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.406328    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hmgx4\""
Jul 31 23:46:53.440102 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.440077    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4"]
Jul 31 23:46:53.495372 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.495343    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h9q9h"]
Jul 31 23:46:53.498844 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:53.498821    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf668c162_e18e_43ab_ad64_d2d290bf1301.slice/crio-8627fd1ae92c5c496c9d8423eb414aa165a536f5ebd568975502c1b1208a1a8f WatchSource:0}: Error finding container 8627fd1ae92c5c496c9d8423eb414aa165a536f5ebd568975502c1b1208a1a8f: Status 404 returned error can't find the container with id 8627fd1ae92c5c496c9d8423eb414aa165a536f5ebd568975502c1b1208a1a8f
Jul 31 23:46:53.529663 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.529640    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/12b03887-d790-428b-9208-3100816be5d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-9nkh4\" (UID: \"12b03887-d790-428b-9208-3100816be5d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4"
Jul 31 23:46:53.536592 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.536562    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:53.630712 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.630677    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/12b03887-d790-428b-9208-3100816be5d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-9nkh4\" (UID: \"12b03887-d790-428b-9208-3100816be5d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4"
Jul 31 23:46:53.630861 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:53.630825    2551 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 31 23:46:53.630910 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:46:53.630893    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12b03887-d790-428b-9208-3100816be5d8-tls-certificates podName:12b03887-d790-428b-9208-3100816be5d8 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:54.130874151 +0000 UTC m=+56.824921564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/12b03887-d790-428b-9208-3100816be5d8-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-9nkh4" (UID: "12b03887-d790-428b-9208-3100816be5d8") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 31 23:46:53.799442 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:53.799349    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:54.136429 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.136338    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/12b03887-d790-428b-9208-3100816be5d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-9nkh4\" (UID: \"12b03887-d790-428b-9208-3100816be5d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4"
Jul 31 23:46:54.139241 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.139217    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/12b03887-d790-428b-9208-3100816be5d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-9nkh4\" (UID: \"12b03887-d790-428b-9208-3100816be5d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4"
Jul 31 23:46:54.230795 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.230759    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h9q9h" event={"ID":"f668c162-e18e-43ab-ad64-d2d290bf1301","Type":"ContainerStarted","Data":"28361e990971a1aed7dc8220650ef87270cba2f6eab4d2e19cde86154310912c"}
Jul 31 23:46:54.230957 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.230832    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h9q9h" event={"ID":"f668c162-e18e-43ab-ad64-d2d290bf1301","Type":"ContainerStarted","Data":"c48355f991de20d9a7f43d4dc6a5444b1477accd6f7392eac1b97f1916b78d7a"}
Jul 31 23:46:54.230957 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.230862    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h9q9h" event={"ID":"f668c162-e18e-43ab-ad64-d2d290bf1301","Type":"ContainerStarted","Data":"8627fd1ae92c5c496c9d8423eb414aa165a536f5ebd568975502c1b1208a1a8f"}
Jul 31 23:46:54.232339 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.232310    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bsrg5" event={"ID":"97fb1913-5741-455f-8595-177262ce365a","Type":"ContainerStarted","Data":"b8952ad861a41826426ab2689e81dc7362226723bac722c12a284e0c92ecbc39"}
Jul 31 23:46:54.238756 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.235797    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl" event={"ID":"03207f34-03d6-4655-ba13-0de638a39a5d","Type":"ContainerStarted","Data":"7777296baa6c7f3eb4f09ecabdb21cda518d3b09b017cfb22b5d346ccf71fcf0"}
Jul 31 23:46:54.238756 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.235877    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl" event={"ID":"03207f34-03d6-4655-ba13-0de638a39a5d","Type":"ContainerStarted","Data":"97788f918203ede0e4eb0aa97e44c9202c1bf68104a06e9219e308a0cdfac91a"}
Jul 31 23:46:54.238756 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.236067    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:46:54.238756 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.238095    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jxsn8" event={"ID":"9ab1356e-7c27-49ae-a3ad-700b2c0ac399","Type":"ContainerStarted","Data":"2fb209953bcccabd016728b627e993aed7b22aec1a50aa298c40d14fc70c9869"}
Jul 31 23:46:54.313425 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.313389    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4"
Jul 31 23:46:54.389345 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.389272    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:54.457283 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.457216    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl" podStartSLOduration=2.4571925009999998 podStartE2EDuration="2.457192501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:54.278010216 +0000 UTC m=+56.972057633" watchObservedRunningTime="2026-07-31 23:46:54.457192501 +0000 UTC m=+57.151239917"
Jul 31 23:46:54.458255 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.458186    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4"]
Jul 31 23:46:54.461975 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:54.461947    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:54.466135 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:46:54.466109    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b03887_d790_428b_9208_3100816be5d8.slice/crio-6da135c39b7cb89aa0daf200bc5a9724a01abc7128c0e81e3dc915b75246334c WatchSource:0}: Error finding container 6da135c39b7cb89aa0daf200bc5a9724a01abc7128c0e81e3dc915b75246334c: Status 404 returned error can't find the container with id 6da135c39b7cb89aa0daf200bc5a9724a01abc7128c0e81e3dc915b75246334c
Jul 31 23:46:55.243213 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:55.243137    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4" event={"ID":"12b03887-d790-428b-9208-3100816be5d8","Type":"ContainerStarted","Data":"6da135c39b7cb89aa0daf200bc5a9724a01abc7128c0e81e3dc915b75246334c"}
Jul 31 23:46:57.250955 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.250896    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h9q9h" event={"ID":"f668c162-e18e-43ab-ad64-d2d290bf1301","Type":"ContainerStarted","Data":"2bb94007effeba6e60df08450322e3f30c8841725e85420c8850e446cf37714e"}
Jul 31 23:46:57.252591 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.252562    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bsrg5" event={"ID":"97fb1913-5741-455f-8595-177262ce365a","Type":"ContainerStarted","Data":"81def70379dd2178b219db7ed6f413c6c6cd9a323aa88bf291367120d15781bf"}
Jul 31 23:46:57.252714 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.252619    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:46:57.252714 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.252640    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bsrg5" event={"ID":"97fb1913-5741-455f-8595-177262ce365a","Type":"ContainerStarted","Data":"d71176db8daa511da169a2f0581b59d511ed5b2ea0cdb1737070bb10eeba320e"}
Jul 31 23:46:57.253979 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.253960    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4" event={"ID":"12b03887-d790-428b-9208-3100816be5d8","Type":"ContainerStarted","Data":"3f0ed3ca593effd63bdddc3dfbc668288e75ba2182f8454927ab1462eba44cb4"}
Jul 31 23:46:57.254051 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.253995    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4"
Jul 31 23:46:57.255495 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.255445    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jxsn8" event={"ID":"9ab1356e-7c27-49ae-a3ad-700b2c0ac399","Type":"ContainerStarted","Data":"e5394a0eab6f60b1ff4159b956f92b527375eb998a9744025859a4591487bbfc"}
Jul 31 23:46:57.258902 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.258884    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4"
Jul 31 23:46:57.273792 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.273756    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h9q9h" podStartSLOduration=2.447635275 podStartE2EDuration="5.27374477s" totalImagesPullingTime="2.826109495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:57.273070837 +0000 UTC m=+59.967118252" watchObservedRunningTime="2026-07-31 23:46:57.27374477 +0000 UTC m=+59.967792164"
Jul 31 23:46:57.290160 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.290120    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bsrg5" podStartSLOduration=2.411645782 podStartE2EDuration="5.290108746s" totalImagesPullingTime="2.878462964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:57.289103287 +0000 UTC m=+59.983150702" watchObservedRunningTime="2026-07-31 23:46:57.290108746 +0000 UTC m=+59.984156162"
Jul 31 23:46:57.310148 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.310107    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-9nkh4" podStartSLOduration=2.313149664 podStartE2EDuration="4.31009491s" totalImagesPullingTime="1.996945246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:57.309931536 +0000 UTC m=+60.003978945" watchObservedRunningTime="2026-07-31 23:46:57.31009491 +0000 UTC m=+60.004142326"
Jul 31 23:46:57.330452 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.329456    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jxsn8" podStartSLOduration=2.555896411 podStartE2EDuration="5.329437568s" totalImagesPullingTime="2.773541157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:57.327475829 +0000 UTC m=+60.021523246" watchObservedRunningTime="2026-07-31 23:46:57.329437568 +0000 UTC m=+60.023484984"
Jul 31 23:46:57.397165 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.397126    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:57.487684 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.487653    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:57.680543 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:57.680470    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:58.176039 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:58.176014    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b47km"
Jul 31 23:46:58.353579 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:58.353531    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:58.445510 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:58.445435    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:59.832147 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:59.832106    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:59.889167 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:46:59.889138    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:00.859074 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:00.859039    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.530739 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:02.530706    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.612554 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:02.612526    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:03.173116 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.173088    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:03.235649 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.235623    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:03.474641 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.474612    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:03.505202 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.505172    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs\") pod \"network-metrics-daemon-r7fbk\" (UID: \"a0d57ecc-0561-4cce-855e-0281fb270d89\") " pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:47:03.507509 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.507492    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:47:03.518297 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.518269    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d57ecc-0561-4cce-855e-0281fb270d89-metrics-certs\") pod \"network-metrics-daemon-r7fbk\" (UID: \"a0d57ecc-0561-4cce-855e-0281fb270d89\") " pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:47:03.572211 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.572179    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:03.605834 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.605807    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:47:03.608137 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.608120    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:47:03.619032 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.619002    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20d38ffc-045d-41cf-bcde-a74d3e0b8361-original-pull-secret\") pod \"global-pull-secret-syncer-hfz7q\" (UID: \"20d38ffc-045d-41cf-bcde-a74d3e0b8361\") " pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:47:03.659768 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.659733    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c2rbb\""
Jul 31 23:47:03.662450 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.662422    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hfz7q"
Jul 31 23:47:03.668008 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.667990    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r7fbk"
Jul 31 23:47:03.706844 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.706804    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9mk\" (UniqueName: \"kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk\") pod \"network-check-target-j2rjc\" (UID: \"15c09ad2-cc88-49b1-b608-6f4bab832087\") " pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:47:03.709214 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.709169    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 23:47:03.720408 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.720198    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 23:47:03.732571 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.732135    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l9mk\" (UniqueName: \"kubernetes.io/projected/15c09ad2-cc88-49b1-b608-6f4bab832087-kube-api-access-2l9mk\") pod \"network-check-target-j2rjc\" (UID: \"15c09ad2-cc88-49b1-b608-6f4bab832087\") " pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:47:03.765069 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.765024    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:03.805137 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.804999    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hfz7q"]
Jul 31 23:47:03.808537 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:47:03.808512    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d38ffc_045d_41cf_bcde_a74d3e0b8361.slice/crio-d83ee397d895dbaa56f0c082bdcd87e31244de2b43e82e26af5321dbbcd1e67d WatchSource:0}: Error finding container d83ee397d895dbaa56f0c082bdcd87e31244de2b43e82e26af5321dbbcd1e67d: Status 404 returned error can't find the container with id d83ee397d895dbaa56f0c082bdcd87e31244de2b43e82e26af5321dbbcd1e67d
Jul 31 23:47:03.824971 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.824315    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-prc8z"]
Jul 31 23:47:03.827252 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.827230    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r7fbk"]
Jul 31 23:47:03.827388 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.827366    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:03.828469 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:47:03.828440    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d57ecc_0561_4cce_855e_0281fb270d89.slice/crio-6aa2374d27e25ec1e62a1f2f40b6c31dadbdb8dac4c3c6ea185c6cf454b6c5b3 WatchSource:0}: Error finding container 6aa2374d27e25ec1e62a1f2f40b6c31dadbdb8dac4c3c6ea185c6cf454b6c5b3: Status 404 returned error can't find the container with id 6aa2374d27e25ec1e62a1f2f40b6c31dadbdb8dac4c3c6ea185c6cf454b6c5b3
Jul 31 23:47:03.829592 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.829572    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 23:47:03.830687 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.830667    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 23:47:03.830781 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.830753    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 23:47:03.831116 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.831034    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f4bpr\""
Jul 31 23:47:03.831116 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.831048    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 23:47:03.831644 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.831627    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 23:47:03.831734 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.831681    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 23:47:03.856731 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.856703    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:03.907936 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.907868    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67tk\" (UniqueName: \"kubernetes.io/projected/a338d921-48df-4ec5-8f68-f4edccb201fd-kube-api-access-j67tk\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:03.907936 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.907911    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-tls\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:03.908139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.907954    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a338d921-48df-4ec5-8f68-f4edccb201fd-root\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:03.908139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.907971    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:03.908139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.907995    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a338d921-48df-4ec5-8f68-f4edccb201fd-metrics-client-ca\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:03.908139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.908008    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a338d921-48df-4ec5-8f68-f4edccb201fd-sys\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:03.908139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.908033    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-textfile\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:03.908139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.908054    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:03.908139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.908074    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-wtmp\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:03.934401 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.934362    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:03.953815 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.953791    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45dqd\""
Jul 31 23:47:03.962555 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:03.962531    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:47:04.009468 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.009119    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-textfile\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.009552 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.009485    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.009552 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.009531    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-wtmp\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.009651 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.009571    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-textfile\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.009806 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.009759    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-wtmp\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.010166 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.010146    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.010414 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.010196    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j67tk\" (UniqueName: \"kubernetes.io/projected/a338d921-48df-4ec5-8f68-f4edccb201fd-kube-api-access-j67tk\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.010414 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.010251    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-tls\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.010539 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.010422    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a338d921-48df-4ec5-8f68-f4edccb201fd-root\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.010539 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:47:04.010360    2551 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 31 23:47:04.010539 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:47:04.010513    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-tls podName:a338d921-48df-4ec5-8f68-f4edccb201fd nodeName:}" failed. No retries permitted until 2026-07-31 23:47:04.510492261 +0000 UTC m=+67.204539669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-tls") pod "node-exporter-prc8z" (UID: "a338d921-48df-4ec5-8f68-f4edccb201fd") : secret "node-exporter-tls" not found
Jul 31 23:47:04.010864 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.010843    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a338d921-48df-4ec5-8f68-f4edccb201fd-root\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.011350 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.010907    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.011437 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.011402    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a338d921-48df-4ec5-8f68-f4edccb201fd-metrics-client-ca\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.011524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.011433    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a338d921-48df-4ec5-8f68-f4edccb201fd-sys\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.011593 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.011579    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a338d921-48df-4ec5-8f68-f4edccb201fd-sys\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.011979 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.011956    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a338d921-48df-4ec5-8f68-f4edccb201fd-metrics-client-ca\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.013337 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.013318    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.020264 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.020238    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67tk\" (UniqueName: \"kubernetes.io/projected/a338d921-48df-4ec5-8f68-f4edccb201fd-kube-api-access-j67tk\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.086641 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.086607    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j2rjc"]
Jul 31 23:47:04.089706 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:47:04.089679    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c09ad2_cc88_49b1_b608_6f4bab832087.slice/crio-6bab3d36d1df796a322a4ab37a40ff9f37e39b4d00d2f882db68aedaa6da5a5c WatchSource:0}: Error finding container 6bab3d36d1df796a322a4ab37a40ff9f37e39b4d00d2f882db68aedaa6da5a5c: Status 404 returned error can't find the container with id 6bab3d36d1df796a322a4ab37a40ff9f37e39b4d00d2f882db68aedaa6da5a5c
Jul 31 23:47:04.092836 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.092812    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:04.167434 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.167393    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:04.269345 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.269251    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:04.279449 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.279415    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hfz7q" event={"ID":"20d38ffc-045d-41cf-bcde-a74d3e0b8361","Type":"ContainerStarted","Data":"d83ee397d895dbaa56f0c082bdcd87e31244de2b43e82e26af5321dbbcd1e67d"}
Jul 31 23:47:04.280781 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.280752    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j2rjc" event={"ID":"15c09ad2-cc88-49b1-b608-6f4bab832087","Type":"ContainerStarted","Data":"6bab3d36d1df796a322a4ab37a40ff9f37e39b4d00d2f882db68aedaa6da5a5c"}
Jul 31 23:47:04.282026 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.282000    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r7fbk" event={"ID":"a0d57ecc-0561-4cce-855e-0281fb270d89","Type":"ContainerStarted","Data":"6aa2374d27e25ec1e62a1f2f40b6c31dadbdb8dac4c3c6ea185c6cf454b6c5b3"}
Jul 31 23:47:04.369753 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.369713    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:04.514052 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.514020    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:04.516204 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.516130    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-tls\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.518758 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.518733    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a338d921-48df-4ec5-8f68-f4edccb201fd-node-exporter-tls\") pod \"node-exporter-prc8z\" (UID: \"a338d921-48df-4ec5-8f68-f4edccb201fd\") " pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.738104 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.737840    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-prc8z"
Jul 31 23:47:04.772298 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.772234    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:04.791631 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:47:04.791591    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda338d921_48df_4ec5_8f68_f4edccb201fd.slice/crio-799051e7a4a2627fa88f98d0bc3620a85508ddf29ac25ea1c0a85781cecebb51 WatchSource:0}: Error finding container 799051e7a4a2627fa88f98d0bc3620a85508ddf29ac25ea1c0a85781cecebb51: Status 404 returned error can't find the container with id 799051e7a4a2627fa88f98d0bc3620a85508ddf29ac25ea1c0a85781cecebb51
Jul 31 23:47:04.867069 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.867030    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:04.868811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.868779    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:47:04.874382 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.872570    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.877862 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.875380    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k2kx5\""
Jul 31 23:47:04.877862 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.875599    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 31 23:47:04.877862 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.875880    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 31 23:47:04.877862 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.876096    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 31 23:47:04.877862 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.876266    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 31 23:47:04.877862 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.876443    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 31 23:47:04.877862 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.876779    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 31 23:47:04.877862 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.876860    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 31 23:47:04.877862 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.877124    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 31 23:47:04.878796 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.878641    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 31 23:47:04.895892 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.895864    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:47:04.919269 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.919150    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.919269 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.919181    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.919269 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.919197    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.919269 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.919214    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-web-config\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.919269 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.919237    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-config-out\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.919554 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.919286    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.919554 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.919317    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfhk8\" (UniqueName: \"kubernetes.io/projected/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-kube-api-access-zfhk8\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.919554 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.919359    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.919554 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.919377    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.919554 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.919463    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.919554 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.919507    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.919760 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.919557    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.919760 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.919593    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:04.981602 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:04.981555    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:05.020118 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020086    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.020278 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020131    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.020383 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020299    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.020383 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020340    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.020383 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020368    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.020383 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020397    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-web-config\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.020694 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020497    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.020694 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020502    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-config-out\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.020694 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020557    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.020694 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020583    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfhk8\" (UniqueName: \"kubernetes.io/projected/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-kube-api-access-zfhk8\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.020694 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020634    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.020694 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:47:05.020674    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-trusted-ca-bundle podName:b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf nodeName:}" failed. No retries permitted until 2026-07-31 23:47:05.52064885 +0000 UTC m=+68.214696248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf") : configmap references non-existent config key: ca-bundle.crt
Jul 31 23:47:05.021072 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020703    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.021072 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020743    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.021072 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.020768    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.021807 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.021743    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.025982 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.025940    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.025982 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.025967    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.026171 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.026149    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-web-config\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.026971 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.026903    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.026971 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.026934    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.027118 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.026990    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.027516 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.027474    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.027591 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.027569    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-config-out\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.029339 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.029309    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.042801 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.042722    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfhk8\" (UniqueName: \"kubernetes.io/projected/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-kube-api-access-zfhk8\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.289422 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.289226    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r7fbk" event={"ID":"a0d57ecc-0561-4cce-855e-0281fb270d89","Type":"ContainerStarted","Data":"e52058f1928a0639e9bf1799e37935914fd902990c7e0d741901ea10d999312b"}
Jul 31 23:47:05.289422 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.289297    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r7fbk" event={"ID":"a0d57ecc-0561-4cce-855e-0281fb270d89","Type":"ContainerStarted","Data":"add200114284a907cc61f3d436a7c0385ae724ada2a44d27431f884e41b396d7"}
Jul 31 23:47:05.290713 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.290653    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prc8z" event={"ID":"a338d921-48df-4ec5-8f68-f4edccb201fd","Type":"ContainerStarted","Data":"799051e7a4a2627fa88f98d0bc3620a85508ddf29ac25ea1c0a85781cecebb51"}
Jul 31 23:47:05.420815 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.420767    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:05.498532 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.498430    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:05.525934 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.525883    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.526973 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.526897    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:05.798399 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:05.798360    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:06.146633 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.146423    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r7fbk" podStartSLOduration=67.158658036 podStartE2EDuration="1m8.146400067s" totalImagesPullingTime="987.742031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:05.306979544 +0000 UTC m=+68.001026958" watchObservedRunningTime="2026-07-31 23:47:06.146400067 +0000 UTC m=+68.840447484"
Jul 31 23:47:06.147158 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.147086    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:47:06.301822 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:47:06.301788    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f2228d_7686_4262_9b9b_ffaa5e9eaaaf.slice/crio-553ea2bfe8c7b01db690985d1d1e06667a1afcf42483311178d4a522a91c427c WatchSource:0}: Error finding container 553ea2bfe8c7b01db690985d1d1e06667a1afcf42483311178d4a522a91c427c: Status 404 returned error can't find the container with id 553ea2bfe8c7b01db690985d1d1e06667a1afcf42483311178d4a522a91c427c
Jul 31 23:47:06.356941 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.356837    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.416329 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.416287    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.717820 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.717740    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.805114 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.805073    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"]
Jul 31 23:47:06.811871 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.811846    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.814396 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.814372    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 31 23:47:06.814523 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.814405    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5q5lc\""
Jul 31 23:47:06.814587 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.814564    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 31 23:47:06.814748 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.814730    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-m2f1s6gdthqk\""
Jul 31 23:47:06.814830 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.814773    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 31 23:47:06.815409 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.815389    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 31 23:47:06.815506 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.815408    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 31 23:47:06.823008 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.822977    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"]
Jul 31 23:47:06.823342 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.823322    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.836636 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.836615    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.836731 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.836650    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.836731 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.836672    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-tls\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.836808 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.836762    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.836808 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.836793    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.836906 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.836828    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a4f7f01-afc4-4b37-9046-856b4573b38e-metrics-client-ca\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.836906 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.836852    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-grpc-tls\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.836906 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.836873    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm2qd\" (UniqueName: \"kubernetes.io/projected/4a4f7f01-afc4-4b37-9046-856b4573b38e-kube-api-access-tm2qd\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.918632 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.918591    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.937609 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.937579    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a4f7f01-afc4-4b37-9046-856b4573b38e-metrics-client-ca\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.937757 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.937634    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-grpc-tls\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.937823 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.937802    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm2qd\" (UniqueName: \"kubernetes.io/projected/4a4f7f01-afc4-4b37-9046-856b4573b38e-kube-api-access-tm2qd\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.937877 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.937858    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.937948 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.937910    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.938014 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.937961    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-tls\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.938068 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.938028    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.938068 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.938047    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.938421 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.938378    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a4f7f01-afc4-4b37-9046-856b4573b38e-metrics-client-ca\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.941090 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.941041    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.941090 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.941086    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-tls\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.941243 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.941125    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.941293 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.941252    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-grpc-tls\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.941451 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.941429    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.941684 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.941663    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4a4f7f01-afc4-4b37-9046-856b4573b38e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:06.946514 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:06.946492    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm2qd\" (UniqueName: \"kubernetes.io/projected/4a4f7f01-afc4-4b37-9046-856b4573b38e-kube-api-access-tm2qd\") pod \"thanos-querier-688b9bd9fb-nxrql\" (UID: \"4a4f7f01-afc4-4b37-9046-856b4573b38e\") " pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:07.018606 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:07.018577    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:07.123132 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:07.123095    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:07.261655 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:07.261324    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bsrg5"
Jul 31 23:47:07.300200 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:07.300099    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerStarted","Data":"553ea2bfe8c7b01db690985d1d1e06667a1afcf42483311178d4a522a91c427c"}
Jul 31 23:47:07.382742 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:07.382705    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:07.483735 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:07.483688    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.203875 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.203840    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6dbb86b874-t4lnc"]
Jul 31 23:47:08.206287 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.206266    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.210345 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.210130    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-52jiqkla8peqm\""
Jul 31 23:47:08.210345 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.210205    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 31 23:47:08.210345 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.210217    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 31 23:47:08.210345 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.210216    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xbcn5\""
Jul 31 23:47:08.210345 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.210280    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 31 23:47:08.210621 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.210459    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 31 23:47:08.216505 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.216476    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dbb86b874-t4lnc"]
Jul 31 23:47:08.251248 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.251217    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/904f772a-9e6f-4284-bb48-969e1f27c06c-secret-metrics-server-client-certs\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.251248 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.251250    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/904f772a-9e6f-4284-bb48-969e1f27c06c-secret-metrics-server-tls\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.251471 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.251269    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkjq5\" (UniqueName: \"kubernetes.io/projected/904f772a-9e6f-4284-bb48-969e1f27c06c-kube-api-access-xkjq5\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.251471 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.251291    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/904f772a-9e6f-4284-bb48-969e1f27c06c-client-ca-bundle\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.251471 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.251364    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/904f772a-9e6f-4284-bb48-969e1f27c06c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.251587 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.251469    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/904f772a-9e6f-4284-bb48-969e1f27c06c-audit-log\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.251587 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.251499    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/904f772a-9e6f-4284-bb48-969e1f27c06c-metrics-server-audit-profiles\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.352593 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.352555    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/904f772a-9e6f-4284-bb48-969e1f27c06c-audit-log\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.352774 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.352611    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/904f772a-9e6f-4284-bb48-969e1f27c06c-metrics-server-audit-profiles\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.352774 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.352642    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/904f772a-9e6f-4284-bb48-969e1f27c06c-secret-metrics-server-client-certs\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.352774 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.352666    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/904f772a-9e6f-4284-bb48-969e1f27c06c-secret-metrics-server-tls\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.352774 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.352689    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkjq5\" (UniqueName: \"kubernetes.io/projected/904f772a-9e6f-4284-bb48-969e1f27c06c-kube-api-access-xkjq5\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.352774 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.352716    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/904f772a-9e6f-4284-bb48-969e1f27c06c-client-ca-bundle\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.352774 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.352747    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/904f772a-9e6f-4284-bb48-969e1f27c06c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.353156 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.353109    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/904f772a-9e6f-4284-bb48-969e1f27c06c-audit-log\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.353528 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.353503    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/904f772a-9e6f-4284-bb48-969e1f27c06c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.353834 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.353814    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/904f772a-9e6f-4284-bb48-969e1f27c06c-metrics-server-audit-profiles\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.355494 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.355470    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/904f772a-9e6f-4284-bb48-969e1f27c06c-secret-metrics-server-tls\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.356136 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.356112    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/904f772a-9e6f-4284-bb48-969e1f27c06c-secret-metrics-server-client-certs\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.356225 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.356164    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/904f772a-9e6f-4284-bb48-969e1f27c06c-client-ca-bundle\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.361657 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.361622    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkjq5\" (UniqueName: \"kubernetes.io/projected/904f772a-9e6f-4284-bb48-969e1f27c06c-kube-api-access-xkjq5\") pod \"metrics-server-6dbb86b874-t4lnc\" (UID: \"904f772a-9e6f-4284-bb48-969e1f27c06c\") " pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.516856 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.516810    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:08.559168 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.559138    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.664913 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.664879    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.772780 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.772716    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.860897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:08.860847    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:09.130857 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.130834    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dbb86b874-t4lnc"]
Jul 31 23:47:09.161960 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.161843    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"]
Jul 31 23:47:09.170702 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:47:09.170672    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904f772a_9e6f_4284_bb48_969e1f27c06c.slice/crio-170f07210d2c0bdb0aff4ed8a6e83a48e6eb23fff179d6f6a7ba7caf0b205947 WatchSource:0}: Error finding container 170f07210d2c0bdb0aff4ed8a6e83a48e6eb23fff179d6f6a7ba7caf0b205947: Status 404 returned error can't find the container with id 170f07210d2c0bdb0aff4ed8a6e83a48e6eb23fff179d6f6a7ba7caf0b205947
Jul 31 23:47:09.218933 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:47:09.218896    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4f7f01_afc4_4b37_9046_856b4573b38e.slice/crio-7230e5d2de67124a7be3e75fb07aa975d99850f9d54253174a1a73673cad0696 WatchSource:0}: Error finding container 7230e5d2de67124a7be3e75fb07aa975d99850f9d54253174a1a73673cad0696: Status 404 returned error can't find the container with id 7230e5d2de67124a7be3e75fb07aa975d99850f9d54253174a1a73673cad0696
Jul 31 23:47:09.309809 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.309722    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j2rjc" event={"ID":"15c09ad2-cc88-49b1-b608-6f4bab832087","Type":"ContainerStarted","Data":"67def65de17b9cf2ec7e24cf04bebddbc2315bfe3be88d59a57c98b240c77ef8"}
Jul 31 23:47:09.310016 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.309858    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:47:09.311341 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.311320    2551 generic.go:356] "Generic (PLEG): container finished" podID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerID="a974e4ce4b7ad607f3749da23df8de4fab99fe358c5a41b115212abfe820e174" exitCode=0
Jul 31 23:47:09.311434 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.311344    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerDied","Data":"a974e4ce4b7ad607f3749da23df8de4fab99fe358c5a41b115212abfe820e174"}
Jul 31 23:47:09.312505 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.312484    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc" event={"ID":"904f772a-9e6f-4284-bb48-969e1f27c06c","Type":"ContainerStarted","Data":"170f07210d2c0bdb0aff4ed8a6e83a48e6eb23fff179d6f6a7ba7caf0b205947"}
Jul 31 23:47:09.313989 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.313962    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hfz7q" event={"ID":"20d38ffc-045d-41cf-bcde-a74d3e0b8361","Type":"ContainerStarted","Data":"6f6bf32aaf910423c284136fff16d8a805d53b66e5cbabf4a796662a582cb4a3"}
Jul 31 23:47:09.315569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.315550    2551 generic.go:356] "Generic (PLEG): container finished" podID="a338d921-48df-4ec5-8f68-f4edccb201fd" containerID="b8aeec3bd375e88df8da2fe17646239c86c50dafa147711865a5830e01a6cd27" exitCode=0
Jul 31 23:47:09.315665 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.315595    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prc8z" event={"ID":"a338d921-48df-4ec5-8f68-f4edccb201fd","Type":"ContainerDied","Data":"b8aeec3bd375e88df8da2fe17646239c86c50dafa147711865a5830e01a6cd27"}
Jul 31 23:47:09.319029 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.319003    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql" event={"ID":"4a4f7f01-afc4-4b37-9046-856b4573b38e","Type":"ContainerStarted","Data":"7230e5d2de67124a7be3e75fb07aa975d99850f9d54253174a1a73673cad0696"}
Jul 31 23:47:09.324287 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.324248    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j2rjc" podStartSLOduration=66.447582737 podStartE2EDuration="1m11.32423577s" totalImagesPullingTime="4.876653033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:09.323793517 +0000 UTC m=+72.017840933" watchObservedRunningTime="2026-07-31 23:47:09.32423577 +0000 UTC m=+72.018283185"
Jul 31 23:47:09.355145 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.355106    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hfz7q" podStartSLOduration=66.196963235 podStartE2EDuration="1m11.355088261s" totalImagesPullingTime="5.158125026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:09.354083627 +0000 UTC m=+72.048131053" watchObservedRunningTime="2026-07-31 23:47:09.355088261 +0000 UTC m=+72.049135677"
Jul 31 23:47:09.760275 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.760238    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:09.866731 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:09.866534    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:10.018689 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.017913    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 23:47:10.021149 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.021112    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.025418 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.025394    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 31 23:47:10.025527 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.025461    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 31 23:47:10.025595 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.025564    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 31 23:47:10.031938 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.026293    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 31 23:47:10.031938 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.026500    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:10.031938 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.026656    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 31 23:47:10.031938 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.027236    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 31 23:47:10.031938 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.027297    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 31 23:47:10.031938 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.027778    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9348n52tu0o1j\""
Jul 31 23:47:10.031938 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.027851    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 31 23:47:10.031938 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.028098    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 31 23:47:10.031938 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.028817    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 31 23:47:10.031938 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.029473    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 31 23:47:10.031938 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.029954    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-c47zk\""
Jul 31 23:47:10.031938 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.030363    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 31 23:47:10.031938 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.030586    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 31 23:47:10.033820 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.033797    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 31 23:47:10.039511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.039396    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 23:47:10.069043 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.068977    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069043 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069029    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069221 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069061    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-config-out\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069221 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069087    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069221 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069119    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069221 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069142    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069221 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069168    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069221 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069197    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069226    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069255    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-web-config\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069279    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069300    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55xzt\" (UniqueName: \"kubernetes.io/projected/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-kube-api-access-55xzt\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069386    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069414    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069464    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-config\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069489    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069518    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069559    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069585    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.069668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.069625    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.170474 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170436    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.170474 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170482    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.170682 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170502    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.170682 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170527    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-web-config\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.170682 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170552    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.170682 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170567    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55xzt\" (UniqueName: \"kubernetes.io/projected/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-kube-api-access-55xzt\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.170682 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170587    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.170682 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170610    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.170682 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170632    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-config\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.170682 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170645    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.170682 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170661    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.171102 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170693    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.171102 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170714    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.171102 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170740    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.171102 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170783    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.171102 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170817    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.171102 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170844    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-config-out\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.171102 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170867    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.171102 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170901    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.171102 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.170937    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.172370 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.171674    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.172477 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.172379    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.172542 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.172504    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.175945 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.175403    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.175945 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.175630    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.178300 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.178237    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.179136 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.179115    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.179491 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.179467    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-config-out\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.179879 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.179832    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.180397 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.180375    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-config\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.181252 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.180855    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-web-config\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.181252 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.181051    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.181252 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.181080    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.181252 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.181215    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.181252 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.181224    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.181560 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.181475    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.182248 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.182207    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.182360 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.182244    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.182595 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.182570    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.183689 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.183645    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55xzt\" (UniqueName: \"kubernetes.io/projected/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-kube-api-access-55xzt\") pod \"prometheus-k8s-0\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.328180 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.328086    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prc8z" event={"ID":"a338d921-48df-4ec5-8f68-f4edccb201fd","Type":"ContainerStarted","Data":"c3d1e587b2df209a659ae1cd29fcde43c400d9197489034f39a83287828cc04a"}
Jul 31 23:47:10.328180 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.328157    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prc8z" event={"ID":"a338d921-48df-4ec5-8f68-f4edccb201fd","Type":"ContainerStarted","Data":"ae44f77a96ac50bace428c5638f5bef45b269cc6442a17b1435af9aebc348a9b"}
Jul 31 23:47:10.344092 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.343808    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:10.349044 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:10.348724    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-prc8z" podStartSLOduration=3.329217929 podStartE2EDuration="7.348705651s" totalImagesPullingTime="3.933159343s" totalInitContainerRuntime="86.328379ms" isStatefulPod=true podCreationTimestamp="2026-07-31 23:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:10.347110545 +0000 UTC m=+73.041157963" watchObservedRunningTime="2026-07-31 23:47:10.348705651 +0000 UTC m=+73.042753069"
Jul 31 23:47:11.686656 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:11.684473    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 23:47:11.695276 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:47:11.695251    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa494f43_6773_4bd9_a9c3_f0e0a06acffd.slice/crio-8c304465218ffcd85a31e39b2e375cbedaf41ed2797f3db4f08912216579ff25 WatchSource:0}: Error finding container 8c304465218ffcd85a31e39b2e375cbedaf41ed2797f3db4f08912216579ff25: Status 404 returned error can't find the container with id 8c304465218ffcd85a31e39b2e375cbedaf41ed2797f3db4f08912216579ff25
Jul 31 23:47:12.337942 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:12.337880    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerStarted","Data":"dd7bd04c1b61d339f3ae56bb0b452fdba51734e4e689e4ecb4fe59722742218c"}
Jul 31 23:47:12.338116 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:12.337971    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerStarted","Data":"0f97be7d5ada580afc2cf92fb61c0804baf08c1f9e63905dd71d2aa8562f07d5"}
Jul 31 23:47:12.338116 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:12.337999    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerStarted","Data":"781d3eb56a9472b032728fb955373d8eaf6f16a5a2016f98d55c66f4e45a318e"}
Jul 31 23:47:12.338116 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:12.338024    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerStarted","Data":"2b6adf82b5fe93c800bb940f1c7f42b9032f168a88f300e220590ce4bb465e59"}
Jul 31 23:47:12.338116 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:12.338048    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerStarted","Data":"6e8d7c6382933b8cf963b1bda0dedf9d056d56008db230d199e551e071b51a62"}
Jul 31 23:47:12.339315 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:12.339288    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc" event={"ID":"904f772a-9e6f-4284-bb48-969e1f27c06c","Type":"ContainerStarted","Data":"9d887e9bde7d5c96c9c7aaf96c0dd46c443fd984f0cdb4baaf5d7965b9d16593"}
Jul 31 23:47:12.340683 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:12.340653    2551 generic.go:356] "Generic (PLEG): container finished" podID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerID="11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136" exitCode=0
Jul 31 23:47:12.340772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:12.340732    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerDied","Data":"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"}
Jul 31 23:47:12.340995 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:12.340801    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerStarted","Data":"8c304465218ffcd85a31e39b2e375cbedaf41ed2797f3db4f08912216579ff25"}
Jul 31 23:47:12.342648 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:12.342625    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql" event={"ID":"4a4f7f01-afc4-4b37-9046-856b4573b38e","Type":"ContainerStarted","Data":"8a2f6ad57e685173fe2634c2677d92382e03aa21ef060c43dde8c19607de736c"}
Jul 31 23:47:12.342757 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:12.342678    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql" event={"ID":"4a4f7f01-afc4-4b37-9046-856b4573b38e","Type":"ContainerStarted","Data":"e40c8d6946cafd33d9c6e5f2fbf45a6ca3ac560de0f3ee52102e2bf4989cc55a"}
Jul 31 23:47:12.342757 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:12.342706    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql" event={"ID":"4a4f7f01-afc4-4b37-9046-856b4573b38e","Type":"ContainerStarted","Data":"b53a299c87cd937d6f1555a0da7b94bc28c90e7e464d2930d46ce582e9156e58"}
Jul 31 23:47:12.401271 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:12.401229    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc" podStartSLOduration=2.033867053 podStartE2EDuration="4.401214171s" totalImagesPullingTime="2.367347118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:12.362600582 +0000 UTC m=+75.056648010" watchObservedRunningTime="2026-07-31 23:47:12.401214171 +0000 UTC m=+75.095261565"
Jul 31 23:47:13.355898 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:13.355854    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerStarted","Data":"3f20de72b3a100855a13677ec7469c837af828e94af3cd104a1b6180bd5365e6"}
Jul 31 23:47:13.409121 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:13.409069    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.619077139 podStartE2EDuration="9.4090489s" totalImagesPullingTime="5.770207019s" totalInitContainerRuntime="19.764742ms" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:13.407694007 +0000 UTC m=+76.101741425" watchObservedRunningTime="2026-07-31 23:47:13.4090489 +0000 UTC m=+76.103096318"
Jul 31 23:47:14.569930 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:14.569890    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:14.672042 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:14.672006    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:15.250042 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:15.249470    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77b58c48d7-pdhnl"
Jul 31 23:47:15.375139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:15.375115    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:15.503370 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:15.503269    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:15.609077 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:15.608523    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:15.697846 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:15.697823    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:16.371400 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.371366    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql" event={"ID":"4a4f7f01-afc4-4b37-9046-856b4573b38e","Type":"ContainerStarted","Data":"32f681626100eee22e874ae2d976aee3aea1e8c899ba813b32e9f64014428fa0"}
Jul 31 23:47:16.371599 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.371447    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql" event={"ID":"4a4f7f01-afc4-4b37-9046-856b4573b38e","Type":"ContainerStarted","Data":"2209b4fe83c7658de4413fe2e03a2c9d6b6b598f551c2e1c07d82f27b3d95a2b"}
Jul 31 23:47:16.371599 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.371494    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:16.371599 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.371530    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql" event={"ID":"4a4f7f01-afc4-4b37-9046-856b4573b38e","Type":"ContainerStarted","Data":"39d3ce74fc24e6792ccf46b093225c472010c051b2b3fbda2479ac3b5ad3cfa2"}
Jul 31 23:47:16.375969 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.375937    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerStarted","Data":"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"}
Jul 31 23:47:16.376093 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.376017    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerStarted","Data":"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"}
Jul 31 23:47:16.376093 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.376067    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerStarted","Data":"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"}
Jul 31 23:47:16.376189 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.376117    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerStarted","Data":"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"}
Jul 31 23:47:16.376189 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.376157    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerStarted","Data":"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"}
Jul 31 23:47:16.376256 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.376205    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerStarted","Data":"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"}
Jul 31 23:47:16.377669 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.377648    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql"
Jul 31 23:47:16.407404 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.407353    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-688b9bd9fb-nxrql" podStartSLOduration=4.511682105 podStartE2EDuration="10.407338737s" totalImagesPullingTime="5.895656632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:16.406529507 +0000 UTC m=+79.100576923" watchObservedRunningTime="2026-07-31 23:47:16.407338737 +0000 UTC m=+79.101386156"
Jul 31 23:47:16.448934 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.448881    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.404844713 podStartE2EDuration="7.448865784s" totalImagesPullingTime="3.020486639s" totalInitContainerRuntime="23.534432ms" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:16.445348223 +0000 UTC m=+79.139395639" watchObservedRunningTime="2026-07-31 23:47:16.448865784 +0000 UTC m=+79.142913201"
Jul 31 23:47:16.477836 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.477802    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:16.554108 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:16.554082    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:17.223563 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:17.223530    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:19.942025 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:19.941991    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:19.995563 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:19.995535    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:20.344146 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:20.344106    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:47:21.863637 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:21.863603    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:21.964547 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:21.964517    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:22.063462 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:22.063420    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:22.164354 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:22.164281    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:22.264170 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:22.264143    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:25.421220 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:25.421179    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:25.523195 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:25.523160    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:27.802127 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:27.802092    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:27.902156 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:27.902124    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:28.004471 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:28.004428    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:28.517718 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:28.517682    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:28.517882 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:28.517746    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:29.064131 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:29.064098    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:29.155388 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:29.155355    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:29.829231 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:29.829198    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:29.933016 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:29.932979    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:40.331092 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:40.331051    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j2rjc"
Jul 31 23:47:41.505282 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:41.505249    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:44.827548 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:44.827486    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:44.888123 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:44.888086    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:48.522817 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:48.522788    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:47:48.527311 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:47:48.527284    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6dbb86b874-t4lnc"
Jul 31 23:48:10.344617 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:10.344569    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:10.364072 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:10.364047    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:10.595364 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:10.595295    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:19.861573 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:19.861535    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:19.919821 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:19.919795    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:24.079982 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.079944    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:48:24.080395 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.080228    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="alertmanager" containerID="cri-o://6e8d7c6382933b8cf963b1bda0dedf9d056d56008db230d199e551e071b51a62" gracePeriod=120
Jul 31 23:48:24.080395 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.080305    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="kube-rbac-proxy-metric" containerID="cri-o://dd7bd04c1b61d339f3ae56bb0b452fdba51734e4e689e4ecb4fe59722742218c" gracePeriod=120
Jul 31 23:48:24.080395 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.080341    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="kube-rbac-proxy-web" containerID="cri-o://781d3eb56a9472b032728fb955373d8eaf6f16a5a2016f98d55c66f4e45a318e" gracePeriod=120
Jul 31 23:48:24.080395 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.080370    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="kube-rbac-proxy" containerID="cri-o://0f97be7d5ada580afc2cf92fb61c0804baf08c1f9e63905dd71d2aa8562f07d5" gracePeriod=120
Jul 31 23:48:24.080641 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.080353    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="prom-label-proxy" containerID="cri-o://3f20de72b3a100855a13677ec7469c837af828e94af3cd104a1b6180bd5365e6" gracePeriod=120
Jul 31 23:48:24.080641 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.080410    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="config-reloader" containerID="cri-o://2b6adf82b5fe93c800bb940f1c7f42b9032f168a88f300e220590ce4bb465e59" gracePeriod=120
Jul 31 23:48:24.633673 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.633630    2551 generic.go:356] "Generic (PLEG): container finished" podID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerID="3f20de72b3a100855a13677ec7469c837af828e94af3cd104a1b6180bd5365e6" exitCode=0
Jul 31 23:48:24.633673 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.633659    2551 generic.go:356] "Generic (PLEG): container finished" podID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerID="dd7bd04c1b61d339f3ae56bb0b452fdba51734e4e689e4ecb4fe59722742218c" exitCode=0
Jul 31 23:48:24.633673 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.633665    2551 generic.go:356] "Generic (PLEG): container finished" podID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerID="0f97be7d5ada580afc2cf92fb61c0804baf08c1f9e63905dd71d2aa8562f07d5" exitCode=0
Jul 31 23:48:24.633673 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.633672    2551 generic.go:356] "Generic (PLEG): container finished" podID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerID="2b6adf82b5fe93c800bb940f1c7f42b9032f168a88f300e220590ce4bb465e59" exitCode=0
Jul 31 23:48:24.633673 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.633679    2551 generic.go:356] "Generic (PLEG): container finished" podID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerID="6e8d7c6382933b8cf963b1bda0dedf9d056d56008db230d199e551e071b51a62" exitCode=0
Jul 31 23:48:24.633969 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.633702    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerDied","Data":"3f20de72b3a100855a13677ec7469c837af828e94af3cd104a1b6180bd5365e6"}
Jul 31 23:48:24.633969 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.633767    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerDied","Data":"dd7bd04c1b61d339f3ae56bb0b452fdba51734e4e689e4ecb4fe59722742218c"}
Jul 31 23:48:24.633969 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.633796    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerDied","Data":"0f97be7d5ada580afc2cf92fb61c0804baf08c1f9e63905dd71d2aa8562f07d5"}
Jul 31 23:48:24.633969 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.633820    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerDied","Data":"2b6adf82b5fe93c800bb940f1c7f42b9032f168a88f300e220590ce4bb465e59"}
Jul 31 23:48:24.633969 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:24.633844    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerDied","Data":"6e8d7c6382933b8cf963b1bda0dedf9d056d56008db230d199e551e071b51a62"}
Jul 31 23:48:25.270019 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.269896    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:25.321001 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.320972    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.395294 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395218    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") "
Jul 31 23:48:25.395294 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395255    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-trusted-ca-bundle\") pod \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") "
Jul 31 23:48:25.395294 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395289    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-config-volume\") pod \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") "
Jul 31 23:48:25.395554 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395374    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-config-out\") pod \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") "
Jul 31 23:48:25.395554 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395451    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-tls-assets\") pod \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") "
Jul 31 23:48:25.395554 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395470    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy-web\") pod \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") "
Jul 31 23:48:25.395554 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395499    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-web-config\") pod \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") "
Jul 31 23:48:25.395554 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395524    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-cluster-tls-config\") pod \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") "
Jul 31 23:48:25.395806 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395566    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-metrics-client-ca\") pod \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") "
Jul 31 23:48:25.395806 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395595    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy\") pod \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") "
Jul 31 23:48:25.395806 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395640    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-main-tls\") pod \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") "
Jul 31 23:48:25.395806 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395673    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfhk8\" (UniqueName: \"kubernetes.io/projected/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-kube-api-access-zfhk8\") pod \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") "
Jul 31 23:48:25.395806 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395689    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:25.395806 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395707    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-main-db\") pod \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\" (UID: \"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf\") "
Jul 31 23:48:25.396159 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.395988    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:25.396159 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.396007    2551 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:25.396611 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.396565    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:48:25.398045 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.398002    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:25.398307 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.398270    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-config-volume" (OuterVolumeSpecName: "config-volume") pod "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:25.399571 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.399420    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:25.399571 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.399473    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-config-out" (OuterVolumeSpecName: "config-out") pod "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:48:25.399571 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.399542    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-kube-api-access-zfhk8" (OuterVolumeSpecName: "kube-api-access-zfhk8") pod "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf"). InnerVolumeSpecName "kube-api-access-zfhk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:48:25.399780 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.399753    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:25.400630 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.400606    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:25.400734 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.400629    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:48:25.403412 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.403276    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:25.410041 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.409908    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-web-config" (OuterVolumeSpecName: "web-config") pod "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" (UID: "b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:25.496765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.496722    2551 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-tls-assets\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:25.496765 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.496760    2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:25.497015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.496775    2551 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-web-config\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:25.497015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.496787    2551 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-cluster-tls-config\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:25.497015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.496801    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-metrics-client-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:25.497015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.496813    2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:25.497015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.496825    2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-main-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:25.497015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.496839    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfhk8\" (UniqueName: \"kubernetes.io/projected/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-kube-api-access-zfhk8\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:25.497015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.496851    2551 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-alertmanager-main-db\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:25.497015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.496863    2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:25.497015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.496874    2551 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-config-volume\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:25.497015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.496886    2551 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf-config-out\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:25.640260 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.640228    2551 generic.go:356] "Generic (PLEG): container finished" podID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerID="781d3eb56a9472b032728fb955373d8eaf6f16a5a2016f98d55c66f4e45a318e" exitCode=0
Jul 31 23:48:25.640427 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.640283    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerDied","Data":"781d3eb56a9472b032728fb955373d8eaf6f16a5a2016f98d55c66f4e45a318e"}
Jul 31 23:48:25.640427 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.640311    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:25.640427 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.640358    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf","Type":"ContainerDied","Data":"553ea2bfe8c7b01db690985d1d1e06667a1afcf42483311178d4a522a91c427c"}
Jul 31 23:48:25.640427 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.640408    2551 scope.go:117] "RemoveContainer" containerID="3f20de72b3a100855a13677ec7469c837af828e94af3cd104a1b6180bd5365e6"
Jul 31 23:48:25.648821 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.648803    2551 scope.go:117] "RemoveContainer" containerID="dd7bd04c1b61d339f3ae56bb0b452fdba51734e4e689e4ecb4fe59722742218c"
Jul 31 23:48:25.655659 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.655641    2551 scope.go:117] "RemoveContainer" containerID="0f97be7d5ada580afc2cf92fb61c0804baf08c1f9e63905dd71d2aa8562f07d5"
Jul 31 23:48:25.662240 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.662219    2551 scope.go:117] "RemoveContainer" containerID="781d3eb56a9472b032728fb955373d8eaf6f16a5a2016f98d55c66f4e45a318e"
Jul 31 23:48:25.664566 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.664543    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:48:25.670500 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.670479    2551 scope.go:117] "RemoveContainer" containerID="2b6adf82b5fe93c800bb940f1c7f42b9032f168a88f300e220590ce4bb465e59"
Jul 31 23:48:25.671116 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.671096    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:48:25.679856 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.679839    2551 scope.go:117] "RemoveContainer" containerID="6e8d7c6382933b8cf963b1bda0dedf9d056d56008db230d199e551e071b51a62"
Jul 31 23:48:25.686740 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.686722    2551 scope.go:117] "RemoveContainer" containerID="a974e4ce4b7ad607f3749da23df8de4fab99fe358c5a41b115212abfe820e174"
Jul 31 23:48:25.693265 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.693248    2551 scope.go:117] "RemoveContainer" containerID="3f20de72b3a100855a13677ec7469c837af828e94af3cd104a1b6180bd5365e6"
Jul 31 23:48:25.693504 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:25.693484    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f20de72b3a100855a13677ec7469c837af828e94af3cd104a1b6180bd5365e6\": container with ID starting with 3f20de72b3a100855a13677ec7469c837af828e94af3cd104a1b6180bd5365e6 not found: ID does not exist" containerID="3f20de72b3a100855a13677ec7469c837af828e94af3cd104a1b6180bd5365e6"
Jul 31 23:48:25.693562 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.693511    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f20de72b3a100855a13677ec7469c837af828e94af3cd104a1b6180bd5365e6"} err="failed to get container status \"3f20de72b3a100855a13677ec7469c837af828e94af3cd104a1b6180bd5365e6\": rpc error: code = NotFound desc = could not find container \"3f20de72b3a100855a13677ec7469c837af828e94af3cd104a1b6180bd5365e6\": container with ID starting with 3f20de72b3a100855a13677ec7469c837af828e94af3cd104a1b6180bd5365e6 not found: ID does not exist"
Jul 31 23:48:25.693562 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.693544    2551 scope.go:117] "RemoveContainer" containerID="dd7bd04c1b61d339f3ae56bb0b452fdba51734e4e689e4ecb4fe59722742218c"
Jul 31 23:48:25.693787 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:25.693769    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd7bd04c1b61d339f3ae56bb0b452fdba51734e4e689e4ecb4fe59722742218c\": container with ID starting with dd7bd04c1b61d339f3ae56bb0b452fdba51734e4e689e4ecb4fe59722742218c not found: ID does not exist" containerID="dd7bd04c1b61d339f3ae56bb0b452fdba51734e4e689e4ecb4fe59722742218c"
Jul 31 23:48:25.693837 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.693791    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd7bd04c1b61d339f3ae56bb0b452fdba51734e4e689e4ecb4fe59722742218c"} err="failed to get container status \"dd7bd04c1b61d339f3ae56bb0b452fdba51734e4e689e4ecb4fe59722742218c\": rpc error: code = NotFound desc = could not find container \"dd7bd04c1b61d339f3ae56bb0b452fdba51734e4e689e4ecb4fe59722742218c\": container with ID starting with dd7bd04c1b61d339f3ae56bb0b452fdba51734e4e689e4ecb4fe59722742218c not found: ID does not exist"
Jul 31 23:48:25.693837 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.693807    2551 scope.go:117] "RemoveContainer" containerID="0f97be7d5ada580afc2cf92fb61c0804baf08c1f9e63905dd71d2aa8562f07d5"
Jul 31 23:48:25.694107 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:25.694086    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f97be7d5ada580afc2cf92fb61c0804baf08c1f9e63905dd71d2aa8562f07d5\": container with ID starting with 0f97be7d5ada580afc2cf92fb61c0804baf08c1f9e63905dd71d2aa8562f07d5 not found: ID does not exist" containerID="0f97be7d5ada580afc2cf92fb61c0804baf08c1f9e63905dd71d2aa8562f07d5"
Jul 31 23:48:25.694181 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.694117    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f97be7d5ada580afc2cf92fb61c0804baf08c1f9e63905dd71d2aa8562f07d5"} err="failed to get container status \"0f97be7d5ada580afc2cf92fb61c0804baf08c1f9e63905dd71d2aa8562f07d5\": rpc error: code = NotFound desc = could not find container \"0f97be7d5ada580afc2cf92fb61c0804baf08c1f9e63905dd71d2aa8562f07d5\": container with ID starting with 0f97be7d5ada580afc2cf92fb61c0804baf08c1f9e63905dd71d2aa8562f07d5 not found: ID does not exist"
Jul 31 23:48:25.694181 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.694139    2551 scope.go:117] "RemoveContainer" containerID="781d3eb56a9472b032728fb955373d8eaf6f16a5a2016f98d55c66f4e45a318e"
Jul 31 23:48:25.694357 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:25.694338    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781d3eb56a9472b032728fb955373d8eaf6f16a5a2016f98d55c66f4e45a318e\": container with ID starting with 781d3eb56a9472b032728fb955373d8eaf6f16a5a2016f98d55c66f4e45a318e not found: ID does not exist" containerID="781d3eb56a9472b032728fb955373d8eaf6f16a5a2016f98d55c66f4e45a318e"
Jul 31 23:48:25.694395 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.694363    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781d3eb56a9472b032728fb955373d8eaf6f16a5a2016f98d55c66f4e45a318e"} err="failed to get container status \"781d3eb56a9472b032728fb955373d8eaf6f16a5a2016f98d55c66f4e45a318e\": rpc error: code = NotFound desc = could not find container \"781d3eb56a9472b032728fb955373d8eaf6f16a5a2016f98d55c66f4e45a318e\": container with ID starting with 781d3eb56a9472b032728fb955373d8eaf6f16a5a2016f98d55c66f4e45a318e not found: ID does not exist"
Jul 31 23:48:25.694395 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.694377    2551 scope.go:117] "RemoveContainer" containerID="2b6adf82b5fe93c800bb940f1c7f42b9032f168a88f300e220590ce4bb465e59"
Jul 31 23:48:25.694575 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:25.694557    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6adf82b5fe93c800bb940f1c7f42b9032f168a88f300e220590ce4bb465e59\": container with ID starting with 2b6adf82b5fe93c800bb940f1c7f42b9032f168a88f300e220590ce4bb465e59 not found: ID does not exist" containerID="2b6adf82b5fe93c800bb940f1c7f42b9032f168a88f300e220590ce4bb465e59"
Jul 31 23:48:25.694635 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.694581    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6adf82b5fe93c800bb940f1c7f42b9032f168a88f300e220590ce4bb465e59"} err="failed to get container status \"2b6adf82b5fe93c800bb940f1c7f42b9032f168a88f300e220590ce4bb465e59\": rpc error: code = NotFound desc = could not find container \"2b6adf82b5fe93c800bb940f1c7f42b9032f168a88f300e220590ce4bb465e59\": container with ID starting with 2b6adf82b5fe93c800bb940f1c7f42b9032f168a88f300e220590ce4bb465e59 not found: ID does not exist"
Jul 31 23:48:25.694635 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.694601    2551 scope.go:117] "RemoveContainer" containerID="6e8d7c6382933b8cf963b1bda0dedf9d056d56008db230d199e551e071b51a62"
Jul 31 23:48:25.694822 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:25.694804    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8d7c6382933b8cf963b1bda0dedf9d056d56008db230d199e551e071b51a62\": container with ID starting with 6e8d7c6382933b8cf963b1bda0dedf9d056d56008db230d199e551e071b51a62 not found: ID does not exist" containerID="6e8d7c6382933b8cf963b1bda0dedf9d056d56008db230d199e551e071b51a62"
Jul 31 23:48:25.694868 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.694825    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8d7c6382933b8cf963b1bda0dedf9d056d56008db230d199e551e071b51a62"} err="failed to get container status \"6e8d7c6382933b8cf963b1bda0dedf9d056d56008db230d199e551e071b51a62\": rpc error: code = NotFound desc = could not find container \"6e8d7c6382933b8cf963b1bda0dedf9d056d56008db230d199e551e071b51a62\": container with ID starting with 6e8d7c6382933b8cf963b1bda0dedf9d056d56008db230d199e551e071b51a62 not found: ID does not exist"
Jul 31 23:48:25.694868 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.694839    2551 scope.go:117] "RemoveContainer" containerID="a974e4ce4b7ad607f3749da23df8de4fab99fe358c5a41b115212abfe820e174"
Jul 31 23:48:25.695050 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:25.695034    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a974e4ce4b7ad607f3749da23df8de4fab99fe358c5a41b115212abfe820e174\": container with ID starting with a974e4ce4b7ad607f3749da23df8de4fab99fe358c5a41b115212abfe820e174 not found: ID does not exist" containerID="a974e4ce4b7ad607f3749da23df8de4fab99fe358c5a41b115212abfe820e174"
Jul 31 23:48:25.695102 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.695052    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a974e4ce4b7ad607f3749da23df8de4fab99fe358c5a41b115212abfe820e174"} err="failed to get container status \"a974e4ce4b7ad607f3749da23df8de4fab99fe358c5a41b115212abfe820e174\": rpc error: code = NotFound desc = could not find container \"a974e4ce4b7ad607f3749da23df8de4fab99fe358c5a41b115212abfe820e174\": container with ID starting with a974e4ce4b7ad607f3749da23df8de4fab99fe358c5a41b115212abfe820e174 not found: ID does not exist"
Jul 31 23:48:25.945641 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:25.945560    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" path="/var/lib/kubelet/pods/b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf/volumes"
Jul 31 23:48:28.332538 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.332500    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 23:48:28.333024 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.332878    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="prometheus" containerID="cri-o://5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703" gracePeriod=600
Jul 31 23:48:28.333024 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.332906    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="kube-rbac-proxy" containerID="cri-o://55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70" gracePeriod=600
Jul 31 23:48:28.333024 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.332948    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="thanos-sidecar" containerID="cri-o://b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912" gracePeriod=600
Jul 31 23:48:28.333024 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.332953    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3" gracePeriod=600
Jul 31 23:48:28.333240 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.333066    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="config-reloader" containerID="cri-o://338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce" gracePeriod=600
Jul 31 23:48:28.333240 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.333068    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="kube-rbac-proxy-web" containerID="cri-o://843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49" gracePeriod=600
Jul 31 23:48:28.564522 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.564494    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:28.654794 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654715    2551 generic.go:356] "Generic (PLEG): container finished" podID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerID="ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3" exitCode=0
Jul 31 23:48:28.654794 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654738    2551 generic.go:356] "Generic (PLEG): container finished" podID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerID="55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70" exitCode=0
Jul 31 23:48:28.654794 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654745    2551 generic.go:356] "Generic (PLEG): container finished" podID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerID="843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49" exitCode=0
Jul 31 23:48:28.654794 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654751    2551 generic.go:356] "Generic (PLEG): container finished" podID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerID="b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912" exitCode=0
Jul 31 23:48:28.654794 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654757    2551 generic.go:356] "Generic (PLEG): container finished" podID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerID="338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce" exitCode=0
Jul 31 23:48:28.654794 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654766    2551 generic.go:356] "Generic (PLEG): container finished" podID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerID="5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703" exitCode=0
Jul 31 23:48:28.654794 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654778    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerDied","Data":"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"}
Jul 31 23:48:28.655214 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654800    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:28.655214 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654852    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerDied","Data":"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"}
Jul 31 23:48:28.655214 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654881    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerDied","Data":"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"}
Jul 31 23:48:28.655214 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654907    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerDied","Data":"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"}
Jul 31 23:48:28.655214 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654928    2551 scope.go:117] "RemoveContainer" containerID="ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"
Jul 31 23:48:28.655214 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654943    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerDied","Data":"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"}
Jul 31 23:48:28.655214 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654968    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerDied","Data":"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"}
Jul 31 23:48:28.655214 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.654993    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa494f43-6773-4bd9-a9c3-f0e0a06acffd","Type":"ContainerDied","Data":"8c304465218ffcd85a31e39b2e375cbedaf41ed2797f3db4f08912216579ff25"}
Jul 31 23:48:28.668504 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.668473    2551 scope.go:117] "RemoveContainer" containerID="55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"
Jul 31 23:48:28.674929 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.674892    2551 scope.go:117] "RemoveContainer" containerID="843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"
Jul 31 23:48:28.681075 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.681056    2551 scope.go:117] "RemoveContainer" containerID="b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"
Jul 31 23:48:28.687079 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.687061    2551 scope.go:117] "RemoveContainer" containerID="338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"
Jul 31 23:48:28.693336 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.693321    2551 scope.go:117] "RemoveContainer" containerID="5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"
Jul 31 23:48:28.700088 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.700068    2551 scope.go:117] "RemoveContainer" containerID="11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"
Jul 31 23:48:28.706019 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.706005    2551 scope.go:117] "RemoveContainer" containerID="ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"
Jul 31 23:48:28.706260 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:28.706238    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3\": container with ID starting with ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3 not found: ID does not exist" containerID="ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"
Jul 31 23:48:28.706299 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.706269    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"} err="failed to get container status \"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3\": rpc error: code = NotFound desc = could not find container \"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3\": container with ID starting with ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3 not found: ID does not exist"
Jul 31 23:48:28.706299 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.706285    2551 scope.go:117] "RemoveContainer" containerID="55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"
Jul 31 23:48:28.706487 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:28.706471    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70\": container with ID starting with 55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70 not found: ID does not exist" containerID="55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"
Jul 31 23:48:28.706525 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.706490    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"} err="failed to get container status \"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70\": rpc error: code = NotFound desc = could not find container \"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70\": container with ID starting with 55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70 not found: ID does not exist"
Jul 31 23:48:28.706525 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.706504    2551 scope.go:117] "RemoveContainer" containerID="843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"
Jul 31 23:48:28.706690 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:28.706675    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49\": container with ID starting with 843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49 not found: ID does not exist" containerID="843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"
Jul 31 23:48:28.706744 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.706692    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"} err="failed to get container status \"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49\": rpc error: code = NotFound desc = could not find container \"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49\": container with ID starting with 843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49 not found: ID does not exist"
Jul 31 23:48:28.706744 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.706704    2551 scope.go:117] "RemoveContainer" containerID="b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"
Jul 31 23:48:28.706895 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:28.706880    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912\": container with ID starting with b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912 not found: ID does not exist" containerID="b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"
Jul 31 23:48:28.706953 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.706900    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"} err="failed to get container status \"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912\": rpc error: code = NotFound desc = could not find container \"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912\": container with ID starting with b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912 not found: ID does not exist"
Jul 31 23:48:28.706953 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.706912    2551 scope.go:117] "RemoveContainer" containerID="338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"
Jul 31 23:48:28.707134 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:28.707120    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce\": container with ID starting with 338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce not found: ID does not exist" containerID="338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"
Jul 31 23:48:28.707183 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.707137    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"} err="failed to get container status \"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce\": rpc error: code = NotFound desc = could not find container \"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce\": container with ID starting with 338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce not found: ID does not exist"
Jul 31 23:48:28.707183 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.707149    2551 scope.go:117] "RemoveContainer" containerID="5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"
Jul 31 23:48:28.707348 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:28.707329    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703\": container with ID starting with 5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703 not found: ID does not exist" containerID="5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"
Jul 31 23:48:28.707397 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.707351    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"} err="failed to get container status \"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703\": rpc error: code = NotFound desc = could not find container \"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703\": container with ID starting with 5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703 not found: ID does not exist"
Jul 31 23:48:28.707397 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.707363    2551 scope.go:117] "RemoveContainer" containerID="11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"
Jul 31 23:48:28.707534 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:48:28.707519    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136\": container with ID starting with 11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136 not found: ID does not exist" containerID="11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"
Jul 31 23:48:28.707574 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.707536    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"} err="failed to get container status \"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136\": rpc error: code = NotFound desc = could not find container \"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136\": container with ID starting with 11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136 not found: ID does not exist"
Jul 31 23:48:28.707574 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.707548    2551 scope.go:117] "RemoveContainer" containerID="ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"
Jul 31 23:48:28.707728 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.707713    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"} err="failed to get container status \"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3\": rpc error: code = NotFound desc = could not find container \"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3\": container with ID starting with ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3 not found: ID does not exist"
Jul 31 23:48:28.707772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.707728    2551 scope.go:117] "RemoveContainer" containerID="55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"
Jul 31 23:48:28.707897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.707881    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"} err="failed to get container status \"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70\": rpc error: code = NotFound desc = could not find container \"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70\": container with ID starting with 55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70 not found: ID does not exist"
Jul 31 23:48:28.707977 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.707897    2551 scope.go:117] "RemoveContainer" containerID="843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"
Jul 31 23:48:28.708080 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.708065    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"} err="failed to get container status \"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49\": rpc error: code = NotFound desc = could not find container \"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49\": container with ID starting with 843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49 not found: ID does not exist"
Jul 31 23:48:28.708121 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.708081    2551 scope.go:117] "RemoveContainer" containerID="b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"
Jul 31 23:48:28.708259 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.708244    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"} err="failed to get container status \"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912\": rpc error: code = NotFound desc = could not find container \"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912\": container with ID starting with b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912 not found: ID does not exist"
Jul 31 23:48:28.708297 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.708259    2551 scope.go:117] "RemoveContainer" containerID="338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"
Jul 31 23:48:28.708428 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.708414    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"} err="failed to get container status \"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce\": rpc error: code = NotFound desc = could not find container \"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce\": container with ID starting with 338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce not found: ID does not exist"
Jul 31 23:48:28.708470 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.708429    2551 scope.go:117] "RemoveContainer" containerID="5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"
Jul 31 23:48:28.708681 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.708660    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"} err="failed to get container status \"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703\": rpc error: code = NotFound desc = could not find container \"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703\": container with ID starting with 5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703 not found: ID does not exist"
Jul 31 23:48:28.708748 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.708684    2551 scope.go:117] "RemoveContainer" containerID="11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"
Jul 31 23:48:28.708889 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.708870    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"} err="failed to get container status \"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136\": rpc error: code = NotFound desc = could not find container \"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136\": container with ID starting with 11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136 not found: ID does not exist"
Jul 31 23:48:28.708952 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.708892    2551 scope.go:117] "RemoveContainer" containerID="ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"
Jul 31 23:48:28.709164 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.709143    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"} err="failed to get container status \"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3\": rpc error: code = NotFound desc = could not find container \"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3\": container with ID starting with ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3 not found: ID does not exist"
Jul 31 23:48:28.709220 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.709164    2551 scope.go:117] "RemoveContainer" containerID="55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"
Jul 31 23:48:28.709379 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.709359    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"} err="failed to get container status \"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70\": rpc error: code = NotFound desc = could not find container \"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70\": container with ID starting with 55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70 not found: ID does not exist"
Jul 31 23:48:28.709425 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.709388    2551 scope.go:117] "RemoveContainer" containerID="843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"
Jul 31 23:48:28.709572 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.709551    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"} err="failed to get container status \"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49\": rpc error: code = NotFound desc = could not find container \"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49\": container with ID starting with 843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49 not found: ID does not exist"
Jul 31 23:48:28.709614 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.709575    2551 scope.go:117] "RemoveContainer" containerID="b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"
Jul 31 23:48:28.709773 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.709756    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"} err="failed to get container status \"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912\": rpc error: code = NotFound desc = could not find container \"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912\": container with ID starting with b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912 not found: ID does not exist"
Jul 31 23:48:28.709811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.709773    2551 scope.go:117] "RemoveContainer" containerID="338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"
Jul 31 23:48:28.710004 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.709987    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"} err="failed to get container status \"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce\": rpc error: code = NotFound desc = could not find container \"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce\": container with ID starting with 338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce not found: ID does not exist"
Jul 31 23:48:28.710053 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.710004    2551 scope.go:117] "RemoveContainer" containerID="5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"
Jul 31 23:48:28.710181 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.710165    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"} err="failed to get container status \"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703\": rpc error: code = NotFound desc = could not find container \"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703\": container with ID starting with 5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703 not found: ID does not exist"
Jul 31 23:48:28.710227 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.710180    2551 scope.go:117] "RemoveContainer" containerID="11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"
Jul 31 23:48:28.710357 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.710335    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"} err="failed to get container status \"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136\": rpc error: code = NotFound desc = could not find container \"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136\": container with ID starting with 11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136 not found: ID does not exist"
Jul 31 23:48:28.710394 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.710358    2551 scope.go:117] "RemoveContainer" containerID="ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"
Jul 31 23:48:28.710574 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.710554    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"} err="failed to get container status \"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3\": rpc error: code = NotFound desc = could not find container \"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3\": container with ID starting with ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3 not found: ID does not exist"
Jul 31 23:48:28.710611 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.710575    2551 scope.go:117] "RemoveContainer" containerID="55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"
Jul 31 23:48:28.710799 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.710781    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"} err="failed to get container status \"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70\": rpc error: code = NotFound desc = could not find container \"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70\": container with ID starting with 55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70 not found: ID does not exist"
Jul 31 23:48:28.710799 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.710798    2551 scope.go:117] "RemoveContainer" containerID="843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"
Jul 31 23:48:28.711017 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.711001    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"} err="failed to get container status \"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49\": rpc error: code = NotFound desc = could not find container \"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49\": container with ID starting with 843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49 not found: ID does not exist"
Jul 31 23:48:28.711065 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.711017    2551 scope.go:117] "RemoveContainer" containerID="b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"
Jul 31 23:48:28.711230 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.711213    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"} err="failed to get container status \"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912\": rpc error: code = NotFound desc = could not find container \"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912\": container with ID starting with b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912 not found: ID does not exist"
Jul 31 23:48:28.711274 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.711231    2551 scope.go:117] "RemoveContainer" containerID="338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"
Jul 31 23:48:28.711431 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.711415    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"} err="failed to get container status \"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce\": rpc error: code = NotFound desc = could not find container \"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce\": container with ID starting with 338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce not found: ID does not exist"
Jul 31 23:48:28.711467 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.711431    2551 scope.go:117] "RemoveContainer" containerID="5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"
Jul 31 23:48:28.711619 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.711605    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"} err="failed to get container status \"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703\": rpc error: code = NotFound desc = could not find container \"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703\": container with ID starting with 5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703 not found: ID does not exist"
Jul 31 23:48:28.711655 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.711619    2551 scope.go:117] "RemoveContainer" containerID="11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"
Jul 31 23:48:28.711814 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.711796    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"} err="failed to get container status \"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136\": rpc error: code = NotFound desc = could not find container \"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136\": container with ID starting with 11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136 not found: ID does not exist"
Jul 31 23:48:28.711849 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.711815    2551 scope.go:117] "RemoveContainer" containerID="ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"
Jul 31 23:48:28.712033 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.712017    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"} err="failed to get container status \"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3\": rpc error: code = NotFound desc = could not find container \"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3\": container with ID starting with ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3 not found: ID does not exist"
Jul 31 23:48:28.712089 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.712033    2551 scope.go:117] "RemoveContainer" containerID="55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"
Jul 31 23:48:28.712250 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.712232    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"} err="failed to get container status \"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70\": rpc error: code = NotFound desc = could not find container \"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70\": container with ID starting with 55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70 not found: ID does not exist"
Jul 31 23:48:28.712294 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.712250    2551 scope.go:117] "RemoveContainer" containerID="843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"
Jul 31 23:48:28.712431 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.712415    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"} err="failed to get container status \"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49\": rpc error: code = NotFound desc = could not find container \"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49\": container with ID starting with 843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49 not found: ID does not exist"
Jul 31 23:48:28.712473 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.712431    2551 scope.go:117] "RemoveContainer" containerID="b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"
Jul 31 23:48:28.712608 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.712589    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"} err="failed to get container status \"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912\": rpc error: code = NotFound desc = could not find container \"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912\": container with ID starting with b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912 not found: ID does not exist"
Jul 31 23:48:28.712658 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.712608    2551 scope.go:117] "RemoveContainer" containerID="338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"
Jul 31 23:48:28.712820 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.712798    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"} err="failed to get container status \"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce\": rpc error: code = NotFound desc = could not find container \"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce\": container with ID starting with 338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce not found: ID does not exist"
Jul 31 23:48:28.712869 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.712821    2551 scope.go:117] "RemoveContainer" containerID="5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"
Jul 31 23:48:28.713063 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.713046    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"} err="failed to get container status \"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703\": rpc error: code = NotFound desc = could not find container \"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703\": container with ID starting with 5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703 not found: ID does not exist"
Jul 31 23:48:28.713135 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.713064    2551 scope.go:117] "RemoveContainer" containerID="11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"
Jul 31 23:48:28.713306 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.713289    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"} err="failed to get container status \"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136\": rpc error: code = NotFound desc = could not find container \"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136\": container with ID starting with 11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136 not found: ID does not exist"
Jul 31 23:48:28.713356 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.713306    2551 scope.go:117] "RemoveContainer" containerID="ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"
Jul 31 23:48:28.713502 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.713485    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3"} err="failed to get container status \"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3\": rpc error: code = NotFound desc = could not find container \"ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3\": container with ID starting with ae8e26e043b028b95efa761a247559e9d933b39aeccf8dbaf30bc61e7ebd63f3 not found: ID does not exist"
Jul 31 23:48:28.713545 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.713502    2551 scope.go:117] "RemoveContainer" containerID="55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"
Jul 31 23:48:28.713719 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.713695    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70"} err="failed to get container status \"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70\": rpc error: code = NotFound desc = could not find container \"55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70\": container with ID starting with 55b7b9a1a7a04a31ad2fd4b22db5289d8971c3f3c021e3d1ff3b262fad3d5b70 not found: ID does not exist"
Jul 31 23:48:28.713789 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.713720    2551 scope.go:117] "RemoveContainer" containerID="843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"
Jul 31 23:48:28.713951 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.713930    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49"} err="failed to get container status \"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49\": rpc error: code = NotFound desc = could not find container \"843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49\": container with ID starting with 843bd2de0a19ead6a4cce5067457ef507877b49a72866dfb39ac3026e0bacd49 not found: ID does not exist"
Jul 31 23:48:28.713995 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.713954    2551 scope.go:117] "RemoveContainer" containerID="b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"
Jul 31 23:48:28.714178 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.714161    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912"} err="failed to get container status \"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912\": rpc error: code = NotFound desc = could not find container \"b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912\": container with ID starting with b2984f38d5342643e7193ec972cef769d2f0b519019d244c30093a5c7f031912 not found: ID does not exist"
Jul 31 23:48:28.714240 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.714179    2551 scope.go:117] "RemoveContainer" containerID="338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"
Jul 31 23:48:28.714388 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.714374    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce"} err="failed to get container status \"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce\": rpc error: code = NotFound desc = could not find container \"338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce\": container with ID starting with 338cf14a29e44f93ed6a38ce0c1d49f60c00dc8b5b638f7ad4636b9bc034cfce not found: ID does not exist"
Jul 31 23:48:28.714425 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.714389    2551 scope.go:117] "RemoveContainer" containerID="5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"
Jul 31 23:48:28.714592 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.714574    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703"} err="failed to get container status \"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703\": rpc error: code = NotFound desc = could not find container \"5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703\": container with ID starting with 5d686b20f3e0f5e94b2231aacae3f4f3a6bf649c88c024341268188f43465703 not found: ID does not exist"
Jul 31 23:48:28.714643 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.714592    2551 scope.go:117] "RemoveContainer" containerID="11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"
Jul 31 23:48:28.714783 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.714768    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136"} err="failed to get container status \"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136\": rpc error: code = NotFound desc = could not find container \"11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136\": container with ID starting with 11aefb36e3e7420ad24f95ac04aa76c7bb9c617117b8c5f3956fdf65d19ff136 not found: ID does not exist"
Jul 31 23:48:28.725054 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725036    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-config\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725109 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725068    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-web-config\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725145 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725107    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55xzt\" (UniqueName: \"kubernetes.io/projected/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-kube-api-access-55xzt\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725224 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725210    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-tls\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725271 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725250    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-1\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725309 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725280    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-thanos-prometheus-http-client-file\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725362 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725314    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-serving-certs-ca-bundle\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725362 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725345    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725459 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725375    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-metrics-client-certs\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725459 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725406    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-grpc-tls\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725459 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725430    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-0\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725620 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725583    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:28.725761 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725705    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:28.725832 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725770    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-metrics-client-ca\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725832 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725803    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-trusted-ca-bundle\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725967 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725843    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-kubelet-serving-ca-bundle\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725967 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725883    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-config-out\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.725967 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.725906    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-db\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.726208 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.726000    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-2\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.726208 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.726028    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-kube-rbac-proxy\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.726208 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.726052    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-tls-assets\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.726208 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.726079    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\" (UID: \"aa494f43-6773-4bd9-a9c3-f0e0a06acffd\") "
Jul 31 23:48:28.726406 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.726222    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:28.726406 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.726346    2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.726406 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.726366    2551 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.726406 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.726383    2551 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-metrics-client-ca\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.727441 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.727411    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:28.728139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.727545    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:28.728139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.727973    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:28.728139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.728011    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:28.728139 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.728114    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:48:28.728439 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.728272    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-kube-api-access-55xzt" (OuterVolumeSpecName: "kube-api-access-55xzt") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "kube-api-access-55xzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:48:28.728439 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.728340    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-config" (OuterVolumeSpecName: "config") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:28.728524 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.728473    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:28.729123 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.729089    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:28.729386 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.729345    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:28.730039 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.730003    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-config-out" (OuterVolumeSpecName: "config-out") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:48:28.730131 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.730058    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:28.730202 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.730170    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:48:28.730202 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.730186    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:28.730445 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.730422    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:28.730860 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.730842    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:28.737604 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.737584    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-web-config" (OuterVolumeSpecName: "web-config") pod "aa494f43-6773-4bd9-a9c3-f0e0a06acffd" (UID: "aa494f43-6773-4bd9-a9c3-f0e0a06acffd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:28.827262 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827227    2551 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827262 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827257    2551 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827262 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827269    2551 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-metrics-client-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827281    2551 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-grpc-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827292    2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827301    2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827310    2551 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827318    2551 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-config-out\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827326    2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-db\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827335    2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827343    2551 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-kube-rbac-proxy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827352    2551 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-tls-assets\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827360    2551 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827369    2551 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-config\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827376    2551 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-web-config\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827385    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55xzt\" (UniqueName: \"kubernetes.io/projected/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-kube-api-access-55xzt\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.827466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.827393    2551 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa494f43-6773-4bd9-a9c3-f0e0a06acffd-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:48:28.977855 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.977820    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 23:48:28.982137 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:28.982108    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 23:48:29.008101 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008067    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 23:48:29.008388 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008370    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="init-config-reloader"
Jul 31 23:48:29.008474 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008392    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="init-config-reloader"
Jul 31 23:48:29.008474 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008406    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="config-reloader"
Jul 31 23:48:29.008474 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008414    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="config-reloader"
Jul 31 23:48:29.008474 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008426    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="kube-rbac-proxy-web"
Jul 31 23:48:29.008474 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008434    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="kube-rbac-proxy-web"
Jul 31 23:48:29.008474 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008444    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="alertmanager"
Jul 31 23:48:29.008474 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008453    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="alertmanager"
Jul 31 23:48:29.008474 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008463    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="kube-rbac-proxy-web"
Jul 31 23:48:29.008474 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008471    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="kube-rbac-proxy-web"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008483    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="kube-rbac-proxy-thanos"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008491    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="kube-rbac-proxy-thanos"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008500    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="config-reloader"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008508    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="config-reloader"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008519    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="prom-label-proxy"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008527    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="prom-label-proxy"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008539    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="prometheus"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008547    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="prometheus"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008558    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="kube-rbac-proxy"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008566    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="kube-rbac-proxy"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008574    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="thanos-sidecar"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008582    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="thanos-sidecar"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008600    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="kube-rbac-proxy-metric"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008609    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="kube-rbac-proxy-metric"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008620    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="init-config-reloader"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008629    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="init-config-reloader"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008638    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="kube-rbac-proxy"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008648    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="kube-rbac-proxy"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008709    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="alertmanager"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008721    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="kube-rbac-proxy-web"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008732    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="prom-label-proxy"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008741    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="config-reloader"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008754    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="kube-rbac-proxy"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008763    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="prometheus"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008771    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="thanos-sidecar"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008779    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="kube-rbac-proxy"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008788    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="config-reloader"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008796    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" containerName="kube-rbac-proxy-thanos"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008804    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="kube-rbac-proxy-web"
Jul 31 23:48:29.008897 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.008813    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f2228d-7686-4262-9b9b-ffaa5e9eaaaf" containerName="kube-rbac-proxy-metric"
Jul 31 23:48:29.012707 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.012686    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.015275 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.015250    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 31 23:48:29.015385 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.015273    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 31 23:48:29.015385 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.015286    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 31 23:48:29.015385 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.015345    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 31 23:48:29.015385 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.015346    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 31 23:48:29.015649 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.015633    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 31 23:48:29.015721 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.015673    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 31 23:48:29.016438 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.016414    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 31 23:48:29.016654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.016442    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 31 23:48:29.016654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.016456    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 31 23:48:29.016654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.016457    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 31 23:48:29.016654 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.016512    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-c47zk\""
Jul 31 23:48:29.016840 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.016733    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9348n52tu0o1j\""
Jul 31 23:48:29.017033 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.017014    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 31 23:48:29.018724 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.018703    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 31 23:48:29.021621 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.021603    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 31 23:48:29.029599 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.029569    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 23:48:29.130162 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130130    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130162 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130163    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130387 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130181    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130387 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130225    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c995521b-63c5-429f-9fee-2760aa423079-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130387 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130256    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130387 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130279    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130387 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130308    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt6k9\" (UniqueName: \"kubernetes.io/projected/c995521b-63c5-429f-9fee-2760aa423079-kube-api-access-kt6k9\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130387 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130334    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c995521b-63c5-429f-9fee-2760aa423079-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130387 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130356    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130597 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130418    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130597 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130442    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130597 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130475    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130597 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130494    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130597 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130527    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-config\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130597 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130561    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130597 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130574    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130604    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-web-config\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130622    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130637    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c995521b-63c5-429f-9fee-2760aa423079-config-out\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.130811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.130656    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.231788 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.231747    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-web-config\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.231997 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.231799    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.231997 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.231826    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c995521b-63c5-429f-9fee-2760aa423079-config-out\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.231997 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.231852    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.231997 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.231889    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.231997 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.231911    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.232272 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232090    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.232457 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232432    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c995521b-63c5-429f-9fee-2760aa423079-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.232531 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232472    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.232531 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232503    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.232531 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232525    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6k9\" (UniqueName: \"kubernetes.io/projected/c995521b-63c5-429f-9fee-2760aa423079-kube-api-access-kt6k9\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.232698 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232551    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c995521b-63c5-429f-9fee-2760aa423079-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.232698 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232577    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.232698 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232610    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.232698 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232636    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.232698 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232649    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.233051 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232710    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.233051 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232765    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.233051 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232796    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.233051 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232829    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-config\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.233051 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232864    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.233051 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.232894    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.233344 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.233331    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c995521b-63c5-429f-9fee-2760aa423079-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.233481 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.233462    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.234173 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.234144    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.234630 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.234603    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c995521b-63c5-429f-9fee-2760aa423079-config-out\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.234949 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.234908    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.235079 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.235054    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.235186 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.235165    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.235352 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.235333    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-web-config\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.235888 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.235865    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.235987 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.235963    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.236201 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.236184    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c995521b-63c5-429f-9fee-2760aa423079-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.236619 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.236597    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.237192 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.237173    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-config\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.237355 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.237335    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.237693 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.237671    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.237841 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.237824    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c995521b-63c5-429f-9fee-2760aa423079-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.238157 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.238142    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c995521b-63c5-429f-9fee-2760aa423079-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.240811 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.240795    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6k9\" (UniqueName: \"kubernetes.io/projected/c995521b-63c5-429f-9fee-2760aa423079-kube-api-access-kt6k9\") pod \"prometheus-k8s-0\" (UID: \"c995521b-63c5-429f-9fee-2760aa423079\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.323020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.322980    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:29.456422 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.456389    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 23:48:29.460085 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:48:29.460047    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc995521b_63c5_429f_9fee_2760aa423079.slice/crio-d44343c190d08115106c939d346d7fa3bafd1d5b8617a95714c0b8e3f622ab4f WatchSource:0}: Error finding container d44343c190d08115106c939d346d7fa3bafd1d5b8617a95714c0b8e3f622ab4f: Status 404 returned error can't find the container with id d44343c190d08115106c939d346d7fa3bafd1d5b8617a95714c0b8e3f622ab4f
Jul 31 23:48:29.661453 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.661422    2551 generic.go:356] "Generic (PLEG): container finished" podID="c995521b-63c5-429f-9fee-2760aa423079" containerID="bb72f6fc5c8eaa3f3d03099ef8886f2d0885a413a96755e27013c6d8d0881609" exitCode=0
Jul 31 23:48:29.661593 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.661483    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c995521b-63c5-429f-9fee-2760aa423079","Type":"ContainerDied","Data":"bb72f6fc5c8eaa3f3d03099ef8886f2d0885a413a96755e27013c6d8d0881609"}
Jul 31 23:48:29.661593 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.661532    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c995521b-63c5-429f-9fee-2760aa423079","Type":"ContainerStarted","Data":"d44343c190d08115106c939d346d7fa3bafd1d5b8617a95714c0b8e3f622ab4f"}
Jul 31 23:48:29.947230 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:29.947193    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa494f43-6773-4bd9-a9c3-f0e0a06acffd" path="/var/lib/kubelet/pods/aa494f43-6773-4bd9-a9c3-f0e0a06acffd/volumes"
Jul 31 23:48:30.668278 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:30.668191    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c995521b-63c5-429f-9fee-2760aa423079","Type":"ContainerStarted","Data":"ce97809da9e089dc94026d407d17355c49704e09b012b5486da5b607ad1fbc59"}
Jul 31 23:48:30.668278 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:30.668275    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c995521b-63c5-429f-9fee-2760aa423079","Type":"ContainerStarted","Data":"e47a93fff75af6b963aa05381afd0076e91c1cfc8262f821eaccb7aca8b1128d"}
Jul 31 23:48:30.668683 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:30.668317    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c995521b-63c5-429f-9fee-2760aa423079","Type":"ContainerStarted","Data":"de0a4cd0c72552e008abfd6a356a2fdc13c2c3e2ed8143eb21878c5c43e97cf5"}
Jul 31 23:48:30.668683 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:30.668356    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c995521b-63c5-429f-9fee-2760aa423079","Type":"ContainerStarted","Data":"3874ab58be95ed99415e32c843eacbe174aa98019ccbd8b689acb996a502a885"}
Jul 31 23:48:30.668683 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:30.668391    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c995521b-63c5-429f-9fee-2760aa423079","Type":"ContainerStarted","Data":"21f338e1b59c78577e5a497780551f938a0e0454f20bce6efdf041097f31ee68"}
Jul 31 23:48:30.668683 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:30.668426    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c995521b-63c5-429f-9fee-2760aa423079","Type":"ContainerStarted","Data":"1e3c83cc725d98f00a2ad62e11ab65c1a0ead7847fbc1695edb4d50812f270eb"}
Jul 31 23:48:30.693278 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:30.693217    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.674250163 podStartE2EDuration="2.693199496s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.949333ms" isStatefulPod=false podCreationTimestamp="2026-07-31 23:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:48:30.691827498 +0000 UTC m=+153.385874925" watchObservedRunningTime="2026-07-31 23:48:30.693199496 +0000 UTC m=+153.387246912"
Jul 31 23:48:31.779143 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:31.779091    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:31.880101 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:31.880054    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:34.323794 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:34.323757    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:48:34.543041 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:34.543006    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:34.598034 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:34.597945    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:44.842984 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:44.842938    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:44.931839 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:44.931803    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:45.059348 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:45.059316    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:45.114815 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:45.114738    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:45.173005 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:48:45.172978    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:04.913894 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:04.913858    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:07.532412 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:07.532379    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:29.323521 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:29.323485    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:49:29.338623 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:29.338598    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:49:29.889543 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:29.889519    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 23:49:32.476936 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:32.476878    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:32.577191 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:32.577156    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:32.678033 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:32.678002    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:54.381878 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:54.381842    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:54.481141 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:54.481101    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:59.311770 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:59.311733    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:59.623287 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:59.623209    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:59.719189 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:59.719152    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:59.811751 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:49:59.811709    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:00.150771 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:00.150738    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:00.272804 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:00.272769    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:00.373958 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:00.373900    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:00.472645 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:00.472618    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:00.676007 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:00.675961    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:00.774390 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:00.774308    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:00.914374 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:00.914343    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:01.013486 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:01.013455    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:01.151730 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:01.151656    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:42.336492 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:42.336460    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:42.388463 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:42.388428    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:53.201425 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:53.201387    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:54.534350 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:54.534313    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:57.735498 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:50:57.735468    2551 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 23:51:04.660871 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:04.660839    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:05.943603 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:05.943561    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:08.473016 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:08.472982    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:08.577156 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:08.577123    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:10.693275 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:10.693243    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:10.807198 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:10.807163    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:13.036936 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.036892    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:13.258092 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.258063    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:13.326739 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.326661    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-t4t5b"]
Jul 31 23:51:13.328660 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.328644    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-t4t5b"
Jul 31 23:51:13.330769 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.330738    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 31 23:51:13.331223 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.331204    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qq8p5\""
Jul 31 23:51:13.331325 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.331213    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 31 23:51:13.337242 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.337220    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-t4t5b"]
Jul 31 23:51:13.459571 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.459534    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:13.507645 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.507610    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb41036b-b290-4aff-a84f-9cef58c4dd0e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-t4t5b\" (UID: \"cb41036b-b290-4aff-a84f-9cef58c4dd0e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t4t5b"
Jul 31 23:51:13.507774 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.507649    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdbc\" (UniqueName: \"kubernetes.io/projected/cb41036b-b290-4aff-a84f-9cef58c4dd0e-kube-api-access-9kdbc\") pod \"cert-manager-webhook-6486ffd997-t4t5b\" (UID: \"cb41036b-b290-4aff-a84f-9cef58c4dd0e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t4t5b"
Jul 31 23:51:13.528338 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.528284    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:13.531140 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.531116    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf"]
Jul 31 23:51:13.533123 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.533104    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf"
Jul 31 23:51:13.535339 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.535319    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-hxj7t\""
Jul 31 23:51:13.546090 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.546071    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf"]
Jul 31 23:51:13.609130 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.609042    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb41036b-b290-4aff-a84f-9cef58c4dd0e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-t4t5b\" (UID: \"cb41036b-b290-4aff-a84f-9cef58c4dd0e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t4t5b"
Jul 31 23:51:13.609130 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.609076    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdbc\" (UniqueName: \"kubernetes.io/projected/cb41036b-b290-4aff-a84f-9cef58c4dd0e-kube-api-access-9kdbc\") pod \"cert-manager-webhook-6486ffd997-t4t5b\" (UID: \"cb41036b-b290-4aff-a84f-9cef58c4dd0e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t4t5b"
Jul 31 23:51:13.616905 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.616871    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb41036b-b290-4aff-a84f-9cef58c4dd0e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-t4t5b\" (UID: \"cb41036b-b290-4aff-a84f-9cef58c4dd0e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t4t5b"
Jul 31 23:51:13.617203 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.617176    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdbc\" (UniqueName: \"kubernetes.io/projected/cb41036b-b290-4aff-a84f-9cef58c4dd0e-kube-api-access-9kdbc\") pod \"cert-manager-webhook-6486ffd997-t4t5b\" (UID: \"cb41036b-b290-4aff-a84f-9cef58c4dd0e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-t4t5b"
Jul 31 23:51:13.652173 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.652145    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-t4t5b"
Jul 31 23:51:13.710517 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.710389    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lw8t\" (UniqueName: \"kubernetes.io/projected/e1230c02-166f-49c5-88d3-108851cd8ecd-kube-api-access-9lw8t\") pod \"cert-manager-cainjector-559cdd8c87-kc8mf\" (UID: \"e1230c02-166f-49c5-88d3-108851cd8ecd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf"
Jul 31 23:51:13.710517 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.710442    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1230c02-166f-49c5-88d3-108851cd8ecd-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kc8mf\" (UID: \"e1230c02-166f-49c5-88d3-108851cd8ecd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf"
Jul 31 23:51:13.772109 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.772079    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-t4t5b"]
Jul 31 23:51:13.775044 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:51:13.775016    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb41036b_b290_4aff_a84f_9cef58c4dd0e.slice/crio-e56a00c81528f1faef0437b11dbdc862a7d4e00a3f0e4280e79126eb666887ff WatchSource:0}: Error finding container e56a00c81528f1faef0437b11dbdc862a7d4e00a3f0e4280e79126eb666887ff: Status 404 returned error can't find the container with id e56a00c81528f1faef0437b11dbdc862a7d4e00a3f0e4280e79126eb666887ff
Jul 31 23:51:13.777056 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.777027    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:51:13.811727 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.811699    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lw8t\" (UniqueName: \"kubernetes.io/projected/e1230c02-166f-49c5-88d3-108851cd8ecd-kube-api-access-9lw8t\") pod \"cert-manager-cainjector-559cdd8c87-kc8mf\" (UID: \"e1230c02-166f-49c5-88d3-108851cd8ecd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf"
Jul 31 23:51:13.811830 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.811731    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1230c02-166f-49c5-88d3-108851cd8ecd-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kc8mf\" (UID: \"e1230c02-166f-49c5-88d3-108851cd8ecd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf"
Jul 31 23:51:13.821689 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.821663    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1230c02-166f-49c5-88d3-108851cd8ecd-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kc8mf\" (UID: \"e1230c02-166f-49c5-88d3-108851cd8ecd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf"
Jul 31 23:51:13.826462 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.826442    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lw8t\" (UniqueName: \"kubernetes.io/projected/e1230c02-166f-49c5-88d3-108851cd8ecd-kube-api-access-9lw8t\") pod \"cert-manager-cainjector-559cdd8c87-kc8mf\" (UID: \"e1230c02-166f-49c5-88d3-108851cd8ecd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf"
Jul 31 23:51:13.841286 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.841268    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf"
Jul 31 23:51:13.961643 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:13.961614    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf"]
Jul 31 23:51:13.964271 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:51:13.964245    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1230c02_166f_49c5_88d3_108851cd8ecd.slice/crio-968fcbca4d1f74c89cb14a21ea1f838da632b4b6cf82e9f24fbbdba994e166fb WatchSource:0}: Error finding container 968fcbca4d1f74c89cb14a21ea1f838da632b4b6cf82e9f24fbbdba994e166fb: Status 404 returned error can't find the container with id 968fcbca4d1f74c89cb14a21ea1f838da632b4b6cf82e9f24fbbdba994e166fb
Jul 31 23:51:14.219259 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:14.219230    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-t4t5b" event={"ID":"cb41036b-b290-4aff-a84f-9cef58c4dd0e","Type":"ContainerStarted","Data":"e56a00c81528f1faef0437b11dbdc862a7d4e00a3f0e4280e79126eb666887ff"}
Jul 31 23:51:14.220301 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:14.220276    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf" event={"ID":"e1230c02-166f-49c5-88d3-108851cd8ecd","Type":"ContainerStarted","Data":"968fcbca4d1f74c89cb14a21ea1f838da632b4b6cf82e9f24fbbdba994e166fb"}
Jul 31 23:51:14.631410 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:14.631299    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:14.755326 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:14.755211    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:17.235595 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:17.235554    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-t4t5b" event={"ID":"cb41036b-b290-4aff-a84f-9cef58c4dd0e","Type":"ContainerStarted","Data":"623f2d13f7d1e4be496dbe9c595a682c077c49846ae05735171faa2fc700491d"}
Jul 31 23:51:17.236020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:17.235655    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-t4t5b"
Jul 31 23:51:17.237256 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:17.237229    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf" event={"ID":"e1230c02-166f-49c5-88d3-108851cd8ecd","Type":"ContainerStarted","Data":"b97b487b9fb31c6775357ca3f3e05d4869b8161de99e25b95e01f24f7da6c86b"}
Jul 31 23:51:17.287511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:17.287450    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-t4t5b" podStartSLOduration=0.923813878 podStartE2EDuration="4.287430221s" totalImagesPullingTime="3.363616343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:51:17.287145166 +0000 UTC m=+319.981192582" watchObservedRunningTime="2026-07-31 23:51:17.287430221 +0000 UTC m=+319.981477628"
Jul 31 23:51:18.072473 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:18.072437    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:18.173782 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:18.173755    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:20.651522 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:20.651482    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:23.248828 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:23.248799    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-t4t5b"
Jul 31 23:51:23.266682 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:23.266634    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kc8mf" podStartSLOduration=7.082975645 podStartE2EDuration="10.266618677s" totalImagesPullingTime="3.183643032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:51:17.322263939 +0000 UTC m=+320.016311355" watchObservedRunningTime="2026-07-31 23:51:23.266618677 +0000 UTC m=+325.960666095"
Jul 31 23:51:26.658718 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:26.658688    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:26.760479 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:26.760446    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:28.692367 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:28.692332    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm"]
Jul 31 23:51:28.695528 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:28.695511    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm"
Jul 31 23:51:28.697627 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:28.697603    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 31 23:51:28.697718 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:28.697651    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:51:28.697718 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:28.697659    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hfh5x\""
Jul 31 23:51:28.702930 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:28.702896    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm"]
Jul 31 23:51:28.721334 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:28.721310    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dszf\" (UniqueName: \"kubernetes.io/projected/737ff00b-2dbb-4d4d-b71a-43d49065c0f8-kube-api-access-2dszf\") pod \"openshift-lws-operator-fd8ccff4c-nmzkm\" (UID: \"737ff00b-2dbb-4d4d-b71a-43d49065c0f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm"
Jul 31 23:51:28.721466 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:28.721358    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/737ff00b-2dbb-4d4d-b71a-43d49065c0f8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nmzkm\" (UID: \"737ff00b-2dbb-4d4d-b71a-43d49065c0f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm"
Jul 31 23:51:28.822302 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:28.822267    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dszf\" (UniqueName: \"kubernetes.io/projected/737ff00b-2dbb-4d4d-b71a-43d49065c0f8-kube-api-access-2dszf\") pod \"openshift-lws-operator-fd8ccff4c-nmzkm\" (UID: \"737ff00b-2dbb-4d4d-b71a-43d49065c0f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm"
Jul 31 23:51:28.822430 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:28.822323    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/737ff00b-2dbb-4d4d-b71a-43d49065c0f8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nmzkm\" (UID: \"737ff00b-2dbb-4d4d-b71a-43d49065c0f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm"
Jul 31 23:51:28.822670 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:28.822655    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/737ff00b-2dbb-4d4d-b71a-43d49065c0f8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nmzkm\" (UID: \"737ff00b-2dbb-4d4d-b71a-43d49065c0f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm"
Jul 31 23:51:28.830205 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:28.830179    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dszf\" (UniqueName: \"kubernetes.io/projected/737ff00b-2dbb-4d4d-b71a-43d49065c0f8-kube-api-access-2dszf\") pod \"openshift-lws-operator-fd8ccff4c-nmzkm\" (UID: \"737ff00b-2dbb-4d4d-b71a-43d49065c0f8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm"
Jul 31 23:51:29.004829 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:29.004797    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm"
Jul 31 23:51:29.129196 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:29.129166    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm"]
Jul 31 23:51:29.132114 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:51:29.132089    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod737ff00b_2dbb_4d4d_b71a_43d49065c0f8.slice/crio-bc8123814c363408b6b324ffad6e29cd7df0c0825a00eeca588956e8473ec5d0 WatchSource:0}: Error finding container bc8123814c363408b6b324ffad6e29cd7df0c0825a00eeca588956e8473ec5d0: Status 404 returned error can't find the container with id bc8123814c363408b6b324ffad6e29cd7df0c0825a00eeca588956e8473ec5d0
Jul 31 23:51:29.286239 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:29.286154    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm" event={"ID":"737ff00b-2dbb-4d4d-b71a-43d49065c0f8","Type":"ContainerStarted","Data":"bc8123814c363408b6b324ffad6e29cd7df0c0825a00eeca588956e8473ec5d0"}
Jul 31 23:51:32.301599 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:32.301561    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm" event={"ID":"737ff00b-2dbb-4d4d-b71a-43d49065c0f8","Type":"ContainerStarted","Data":"1608d554dd66275b4d489697030b0f1772e3f3110fb68491ea0b29fabe526cdd"}
Jul 31 23:51:32.317837 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:32.317788    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nmzkm" podStartSLOduration=1.8020256749999999 podStartE2EDuration="4.317773956s" totalImagesPullingTime="2.515748281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:51:32.317008506 +0000 UTC m=+335.011055936" watchObservedRunningTime="2026-07-31 23:51:32.317773956 +0000 UTC m=+335.011821371"
Jul 31 23:51:37.901365 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:37.901332    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:42.682522 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:42.682485    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:42.773156 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:42.773112    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:43.703988 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:43.703954    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:46.720221 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:46.720186    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:46.832722 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:46.832689    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:49.402740 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.402711    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"]
Jul 31 23:51:49.406885 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.406864    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.410173 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.410149    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 31 23:51:49.410311 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.410175    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 23:51:49.410311 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.410252    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 31 23:51:49.410392 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.410308    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-vjb5w\""
Jul 31 23:51:49.432754 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.432731    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"]
Jul 31 23:51:49.489731 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.489697    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.489731 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.489731    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.489950 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.489753    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.489950 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.489787    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.489950 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.489822    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.489950 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.489864    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.489950 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.489882    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.489950 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.489899    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94hs6\" (UniqueName: \"kubernetes.io/projected/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-kube-api-access-94hs6\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.490182 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.489981    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.507226 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.507201    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:49.590568 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.590525    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.590744 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.590587    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.590744 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.590605    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.590744 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.590622    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94hs6\" (UniqueName: \"kubernetes.io/projected/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-kube-api-access-94hs6\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.590744 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.590661    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.590744 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.590682    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.591038 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.590788    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.591038 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.590844    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.591038 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.590874    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.591038 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.591005    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.591243 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.591076    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.591243 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.591088    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.591621 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.591593    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.591733 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.591647    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.592967 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.592947    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.593202 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.593186    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.612136 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.612109    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.616889 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.616867    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94hs6\" (UniqueName: \"kubernetes.io/projected/cbeff71f-8a02-4ed2-94cd-aa1ced7671ac-kube-api-access-94hs6\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7\" (UID: \"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.617900 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.617871    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:49.716569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.716494    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:49.872126 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:49.872095    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"]
Jul 31 23:51:49.875166 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:51:49.875134    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbeff71f_8a02_4ed2_94cd_aa1ced7671ac.slice/crio-8e6a400bdbc511d5fd29b3e6b497958aa322bb21de3aeb866f8f545886b324a3 WatchSource:0}: Error finding container 8e6a400bdbc511d5fd29b3e6b497958aa322bb21de3aeb866f8f545886b324a3: Status 404 returned error can't find the container with id 8e6a400bdbc511d5fd29b3e6b497958aa322bb21de3aeb866f8f545886b324a3
Jul 31 23:51:50.366065 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:50.365959    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7" event={"ID":"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac","Type":"ContainerStarted","Data":"8e6a400bdbc511d5fd29b3e6b497958aa322bb21de3aeb866f8f545886b324a3"}
Jul 31 23:51:50.978027 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:50.977958    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:51.063630 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.063596    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"]
Jul 31 23:51:51.065793 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.065760    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.069658 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.069632    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 31 23:51:51.069794 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.069683    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 31 23:51:51.069979 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.069958    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 31 23:51:51.070089 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.070070    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tlh7w\""
Jul 31 23:51:51.101312 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.101287    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"]
Jul 31 23:51:51.112043 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.112010    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:51.204482 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.204444    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvrv\" (UniqueName: \"kubernetes.io/projected/cc5d85bc-955d-4cfa-88a4-7d036c8ffe20-kube-api-access-grvrv\") pod \"lws-controller-manager-6cb8c69f9-4t7n9\" (UID: \"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.204668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.204499    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc5d85bc-955d-4cfa-88a4-7d036c8ffe20-cert\") pod \"lws-controller-manager-6cb8c69f9-4t7n9\" (UID: \"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.204668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.204564    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cc5d85bc-955d-4cfa-88a4-7d036c8ffe20-manager-config\") pod \"lws-controller-manager-6cb8c69f9-4t7n9\" (UID: \"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.204668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.204593    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc5d85bc-955d-4cfa-88a4-7d036c8ffe20-metrics-cert\") pod \"lws-controller-manager-6cb8c69f9-4t7n9\" (UID: \"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.305815 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.305735    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvrv\" (UniqueName: \"kubernetes.io/projected/cc5d85bc-955d-4cfa-88a4-7d036c8ffe20-kube-api-access-grvrv\") pod \"lws-controller-manager-6cb8c69f9-4t7n9\" (UID: \"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.305815 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.305784    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc5d85bc-955d-4cfa-88a4-7d036c8ffe20-cert\") pod \"lws-controller-manager-6cb8c69f9-4t7n9\" (UID: \"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.306072 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.305839    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cc5d85bc-955d-4cfa-88a4-7d036c8ffe20-manager-config\") pod \"lws-controller-manager-6cb8c69f9-4t7n9\" (UID: \"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.306072 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.305864    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc5d85bc-955d-4cfa-88a4-7d036c8ffe20-metrics-cert\") pod \"lws-controller-manager-6cb8c69f9-4t7n9\" (UID: \"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.306669 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.306626    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cc5d85bc-955d-4cfa-88a4-7d036c8ffe20-manager-config\") pod \"lws-controller-manager-6cb8c69f9-4t7n9\" (UID: \"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.308573 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.308547    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc5d85bc-955d-4cfa-88a4-7d036c8ffe20-metrics-cert\") pod \"lws-controller-manager-6cb8c69f9-4t7n9\" (UID: \"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.308697 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.308651    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc5d85bc-955d-4cfa-88a4-7d036c8ffe20-cert\") pod \"lws-controller-manager-6cb8c69f9-4t7n9\" (UID: \"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.321496 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.321449    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvrv\" (UniqueName: \"kubernetes.io/projected/cc5d85bc-955d-4cfa-88a4-7d036c8ffe20-kube-api-access-grvrv\") pod \"lws-controller-manager-6cb8c69f9-4t7n9\" (UID: \"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20\") " pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.378157 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.378128    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:51.556879 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:51.556837    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"]
Jul 31 23:51:51.559354 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:51:51.559313    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5d85bc_955d_4cfa_88a4_7d036c8ffe20.slice/crio-a9144d746f1052df50f13286c97aee4daa88ccaed9838c3920f5416ae7e3242d WatchSource:0}: Error finding container a9144d746f1052df50f13286c97aee4daa88ccaed9838c3920f5416ae7e3242d: Status 404 returned error can't find the container with id a9144d746f1052df50f13286c97aee4daa88ccaed9838c3920f5416ae7e3242d
Jul 31 23:51:52.376681 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:52.376636    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9" event={"ID":"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20","Type":"ContainerStarted","Data":"a9144d746f1052df50f13286c97aee4daa88ccaed9838c3920f5416ae7e3242d"}
Jul 31 23:51:52.614464 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:52.614411    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:53.787384 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:53.787352    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 23:51:53.787690 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:53.787423    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 23:51:53.787690 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:53.787454    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 23:51:54.385424 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:54.385389    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7" event={"ID":"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac","Type":"ContainerStarted","Data":"f89eb622de866391c1722b4a6c3d16dba205da322f033094a99a0e1593dbf68a"}
Jul 31 23:51:54.386767 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:54.386741    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9" event={"ID":"cc5d85bc-955d-4cfa-88a4-7d036c8ffe20","Type":"ContainerStarted","Data":"2f798ccb723f01f8dc799ec6ddc3eae108bc90b9def764d039d599254d10c77b"}
Jul 31 23:51:54.386885 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:54.386857    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:51:54.424229 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:54.424186    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7" podStartSLOduration=1.514172731 podStartE2EDuration="5.424171853s" totalImagesPullingTime="3.909999122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:51:54.41844926 +0000 UTC m=+357.112496675" watchObservedRunningTime="2026-07-31 23:51:54.424171853 +0000 UTC m=+357.118219268"
Jul 31 23:51:54.448734 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:54.448686    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9" podStartSLOduration=1.22352191 podStartE2EDuration="3.448669817s" totalImagesPullingTime="2.225147907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:51:54.446991705 +0000 UTC m=+357.141039120" watchObservedRunningTime="2026-07-31 23:51:54.448669817 +0000 UTC m=+357.142717232"
Jul 31 23:51:54.717236 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:54.717145    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:54.721595 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:54.721565    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:55.390273 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:55.390244    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:55.391261 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:55.391243    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:51:55.745511 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:55.745479    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:55.903043 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:51:55.903003    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:05.392549 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:05.392517    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cb8c69f9-4t7n9"
Jul 31 23:52:05.995935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:05.995880    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:06.134523 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:06.134482    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:10.531725 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:10.531671    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:12.917701 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:12.917661    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:13.326930 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:13.326896    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:14.750824 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:14.750790    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:14.849849 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:14.849785    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:15.932531 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:15.932500    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:16.180982 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:16.180946    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:17.135568 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:17.135529    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:17.233659 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:17.233621    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:17.336703 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:17.336672    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:17.588076 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:17.588037    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:18.122250 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:18.122221    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:18.419635 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:18.419553    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:18.524841 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:18.524809    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:18.626871 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:18.626839    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:18.940172 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:18.940137    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:19.081711 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:19.081680    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:20.500940 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:20.500894    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:20.600703 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:20.600669    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:23.930707 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:23.930670    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:24.032716 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:24.032681    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:28.594970 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:28.594937    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:28.697573 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:28.697535    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:29.558368 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.558331    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"]
Jul 31 23:52:29.562333 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.562305    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:29.568356 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.568334    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"
Jul 31 23:52:29.570578 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.570554    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 31 23:52:29.570679 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.570596    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"]
Jul 31 23:52:29.570770 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.570749    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 31 23:52:29.571236 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.571219    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 31 23:52:29.571236 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.571229    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wd7gf\""
Jul 31 23:52:29.571356 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.571280    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 31 23:52:29.621968 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.621940    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e174ef7-e0d1-4ea4-be22-062227293fde-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ckq7h\" (UID: \"1e174ef7-e0d1-4ea4-be22-062227293fde\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"
Jul 31 23:52:29.622305 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.621973    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lft57\" (UniqueName: \"kubernetes.io/projected/1e174ef7-e0d1-4ea4-be22-062227293fde-kube-api-access-lft57\") pod \"kuadrant-console-plugin-598f86468b-ckq7h\" (UID: \"1e174ef7-e0d1-4ea4-be22-062227293fde\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"
Jul 31 23:52:29.622305 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.622008    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e174ef7-e0d1-4ea4-be22-062227293fde-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-ckq7h\" (UID: \"1e174ef7-e0d1-4ea4-be22-062227293fde\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"
Jul 31 23:52:29.723357 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.723325    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e174ef7-e0d1-4ea4-be22-062227293fde-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ckq7h\" (UID: \"1e174ef7-e0d1-4ea4-be22-062227293fde\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"
Jul 31 23:52:29.723542 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.723365    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lft57\" (UniqueName: \"kubernetes.io/projected/1e174ef7-e0d1-4ea4-be22-062227293fde-kube-api-access-lft57\") pod \"kuadrant-console-plugin-598f86468b-ckq7h\" (UID: \"1e174ef7-e0d1-4ea4-be22-062227293fde\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"
Jul 31 23:52:29.723542 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.723401    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e174ef7-e0d1-4ea4-be22-062227293fde-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-ckq7h\" (UID: \"1e174ef7-e0d1-4ea4-be22-062227293fde\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"
Jul 31 23:52:29.723542 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:52:29.723478    2551 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 31 23:52:29.723712 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:52:29.723552    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e174ef7-e0d1-4ea4-be22-062227293fde-plugin-serving-cert podName:1e174ef7-e0d1-4ea4-be22-062227293fde nodeName:}" failed. No retries permitted until 2026-07-31 23:52:30.223533942 +0000 UTC m=+392.917581349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/1e174ef7-e0d1-4ea4-be22-062227293fde-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-ckq7h" (UID: "1e174ef7-e0d1-4ea4-be22-062227293fde") : secret "plugin-serving-cert" not found
Jul 31 23:52:29.724079 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.724057    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e174ef7-e0d1-4ea4-be22-062227293fde-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-ckq7h\" (UID: \"1e174ef7-e0d1-4ea4-be22-062227293fde\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"
Jul 31 23:52:29.731246 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:29.731226    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lft57\" (UniqueName: \"kubernetes.io/projected/1e174ef7-e0d1-4ea4-be22-062227293fde-kube-api-access-lft57\") pod \"kuadrant-console-plugin-598f86468b-ckq7h\" (UID: \"1e174ef7-e0d1-4ea4-be22-062227293fde\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"
Jul 31 23:52:30.105625 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:30.105593    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:30.228345 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:30.228310    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e174ef7-e0d1-4ea4-be22-062227293fde-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ckq7h\" (UID: \"1e174ef7-e0d1-4ea4-be22-062227293fde\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"
Jul 31 23:52:30.230776 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:30.230750    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e174ef7-e0d1-4ea4-be22-062227293fde-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ckq7h\" (UID: \"1e174ef7-e0d1-4ea4-be22-062227293fde\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"
Jul 31 23:52:30.479211 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:30.479183    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"
Jul 31 23:52:30.604208 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:30.604163    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h"]
Jul 31 23:52:30.606960 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:52:30.606929    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e174ef7_e0d1_4ea4_be22_062227293fde.slice/crio-9bdf295ffcc34a4752c358c4f7bf90a3076ccdc918bb12e6a1752545b0fd1462 WatchSource:0}: Error finding container 9bdf295ffcc34a4752c358c4f7bf90a3076ccdc918bb12e6a1752545b0fd1462: Status 404 returned error can't find the container with id 9bdf295ffcc34a4752c358c4f7bf90a3076ccdc918bb12e6a1752545b0fd1462
Jul 31 23:52:31.528611 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:31.528573    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h" event={"ID":"1e174ef7-e0d1-4ea4-be22-062227293fde","Type":"ContainerStarted","Data":"9bdf295ffcc34a4752c358c4f7bf90a3076ccdc918bb12e6a1752545b0fd1462"}
Jul 31 23:52:31.548877 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:31.548835    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:31.650663 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:31.650610    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:34.805388 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:34.805340    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:35.553516 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:35.553434    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h" event={"ID":"1e174ef7-e0d1-4ea4-be22-062227293fde","Type":"ContainerStarted","Data":"1d5949eced2bb06e3aca0c4b5c9c4ae176a2dae7293a7365b091fb1ad6380014"}
Jul 31 23:52:35.568578 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:35.568533    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ckq7h" podStartSLOduration=1.941130767 podStartE2EDuration="6.568517467s" totalImagesPullingTime="4.6273867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:35.567195214 +0000 UTC m=+398.261242638" watchObservedRunningTime="2026-07-31 23:52:35.568517467 +0000 UTC m=+398.262564882"
Jul 31 23:52:36.385633 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:36.385555    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:36.478932 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:36.478884    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:36.548088 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:36.548054    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:42.998014 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:42.997977    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:43.101841 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:43.101805    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:48.431872 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:48.431832    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:48.558138 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:48.558103    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:48.787307 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:48.787270    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:48.843483 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:48.843449    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:51.452766 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:51.452732    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:51.508796 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:51.508768    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:52.190510 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:52.190473    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:52.346288 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:52.346252    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:55.957202 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:55.957169    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:56.052164 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:56.052128    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:56.114077 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:56.114043    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:59.139457 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:52:59.139424    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:13.290368 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.290337    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"]
Jul 31 23:53:13.294456 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.294425    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:13.309279 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.309258    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"]
Jul 31 23:53:13.309384 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.309361    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"
Jul 31 23:53:13.311488 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.311470    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 31 23:53:13.392606 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.392572    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"]
Jul 31 23:53:13.393503 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.393482    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7hql\" (UniqueName: \"kubernetes.io/projected/c4f8480a-023e-4860-b516-4be9f59e61b2-kube-api-access-t7hql\") pod \"limitador-limitador-859c4d96d6-qcvgm\" (UID: \"c4f8480a-023e-4860-b516-4be9f59e61b2\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"
Jul 31 23:53:13.393566 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.393552    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4f8480a-023e-4860-b516-4be9f59e61b2-config-file\") pod \"limitador-limitador-859c4d96d6-qcvgm\" (UID: \"c4f8480a-023e-4860-b516-4be9f59e61b2\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"
Jul 31 23:53:13.394259 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.394226    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:13.493847 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.493810    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:13.494053 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.493956    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4f8480a-023e-4860-b516-4be9f59e61b2-config-file\") pod \"limitador-limitador-859c4d96d6-qcvgm\" (UID: \"c4f8480a-023e-4860-b516-4be9f59e61b2\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"
Jul 31 23:53:13.494121 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.494056    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7hql\" (UniqueName: \"kubernetes.io/projected/c4f8480a-023e-4860-b516-4be9f59e61b2-kube-api-access-t7hql\") pod \"limitador-limitador-859c4d96d6-qcvgm\" (UID: \"c4f8480a-023e-4860-b516-4be9f59e61b2\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"
Jul 31 23:53:13.494583 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.494557    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4f8480a-023e-4860-b516-4be9f59e61b2-config-file\") pod \"limitador-limitador-859c4d96d6-qcvgm\" (UID: \"c4f8480a-023e-4860-b516-4be9f59e61b2\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"
Jul 31 23:53:13.501404 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.501382    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7hql\" (UniqueName: \"kubernetes.io/projected/c4f8480a-023e-4860-b516-4be9f59e61b2-kube-api-access-t7hql\") pod \"limitador-limitador-859c4d96d6-qcvgm\" (UID: \"c4f8480a-023e-4860-b516-4be9f59e61b2\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"
Jul 31 23:53:13.619171 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.619066    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"
Jul 31 23:53:13.743509 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:13.743474    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"]
Jul 31 23:53:13.747644 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:53:13.747616    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f8480a_023e_4860_b516_4be9f59e61b2.slice/crio-930f82d275f2373ded25f5fef6927072f02ef23d1f38c47b03497e5820ca69fd WatchSource:0}: Error finding container 930f82d275f2373ded25f5fef6927072f02ef23d1f38c47b03497e5820ca69fd: Status 404 returned error can't find the container with id 930f82d275f2373ded25f5fef6927072f02ef23d1f38c47b03497e5820ca69fd
Jul 31 23:53:14.124077 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:14.124043    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:14.185020 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:14.184989    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:14.703130 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:14.703070    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm" event={"ID":"c4f8480a-023e-4860-b516-4be9f59e61b2","Type":"ContainerStarted","Data":"930f82d275f2373ded25f5fef6927072f02ef23d1f38c47b03497e5820ca69fd"}
Jul 31 23:53:16.713217 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:16.713180    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm" event={"ID":"c4f8480a-023e-4860-b516-4be9f59e61b2","Type":"ContainerStarted","Data":"50216cbd38fc87689f3eea41f49c61f7c967b359d30777d6b80a0f7bc7096547"}
Jul 31 23:53:16.713660 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:16.713250    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"
Jul 31 23:53:16.729084 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:16.729005    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm" podStartSLOduration=1.3832914170000001 podStartE2EDuration="3.728991248s" totalImagesPullingTime="2.345699831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:53:16.727683713 +0000 UTC m=+439.421731126" watchObservedRunningTime="2026-07-31 23:53:16.728991248 +0000 UTC m=+439.423038697"
Jul 31 23:53:27.718105 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:27.718015    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"
Jul 31 23:53:27.847602 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:27.847571    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"]
Jul 31 23:53:27.847787 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:27.847763    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm" podUID="c4f8480a-023e-4860-b516-4be9f59e61b2" containerName="limitador" containerID="cri-o://50216cbd38fc87689f3eea41f49c61f7c967b359d30777d6b80a0f7bc7096547" gracePeriod=30
Jul 31 23:53:28.389218 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.389194    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"
Jul 31 23:53:28.423826 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.423759    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4f8480a-023e-4860-b516-4be9f59e61b2-config-file\") pod \"c4f8480a-023e-4860-b516-4be9f59e61b2\" (UID: \"c4f8480a-023e-4860-b516-4be9f59e61b2\") "
Jul 31 23:53:28.423826 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.423806    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7hql\" (UniqueName: \"kubernetes.io/projected/c4f8480a-023e-4860-b516-4be9f59e61b2-kube-api-access-t7hql\") pod \"c4f8480a-023e-4860-b516-4be9f59e61b2\" (UID: \"c4f8480a-023e-4860-b516-4be9f59e61b2\") "
Jul 31 23:53:28.424194 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.424168    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f8480a-023e-4860-b516-4be9f59e61b2-config-file" (OuterVolumeSpecName: "config-file") pod "c4f8480a-023e-4860-b516-4be9f59e61b2" (UID: "c4f8480a-023e-4860-b516-4be9f59e61b2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:53:28.426350 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.426319    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f8480a-023e-4860-b516-4be9f59e61b2-kube-api-access-t7hql" (OuterVolumeSpecName: "kube-api-access-t7hql") pod "c4f8480a-023e-4860-b516-4be9f59e61b2" (UID: "c4f8480a-023e-4860-b516-4be9f59e61b2"). InnerVolumeSpecName "kube-api-access-t7hql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:53:28.525224 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.525197    2551 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4f8480a-023e-4860-b516-4be9f59e61b2-config-file\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:53:28.525224 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.525221    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7hql\" (UniqueName: \"kubernetes.io/projected/c4f8480a-023e-4860-b516-4be9f59e61b2-kube-api-access-t7hql\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:53:28.761599 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.761561    2551 generic.go:356] "Generic (PLEG): container finished" podID="c4f8480a-023e-4860-b516-4be9f59e61b2" containerID="50216cbd38fc87689f3eea41f49c61f7c967b359d30777d6b80a0f7bc7096547" exitCode=0
Jul 31 23:53:28.762015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.761615    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"
Jul 31 23:53:28.762015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.761647    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm" event={"ID":"c4f8480a-023e-4860-b516-4be9f59e61b2","Type":"ContainerDied","Data":"50216cbd38fc87689f3eea41f49c61f7c967b359d30777d6b80a0f7bc7096547"}
Jul 31 23:53:28.762015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.761704    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-qcvgm" event={"ID":"c4f8480a-023e-4860-b516-4be9f59e61b2","Type":"ContainerDied","Data":"930f82d275f2373ded25f5fef6927072f02ef23d1f38c47b03497e5820ca69fd"}
Jul 31 23:53:28.762015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.761729    2551 scope.go:117] "RemoveContainer" containerID="50216cbd38fc87689f3eea41f49c61f7c967b359d30777d6b80a0f7bc7096547"
Jul 31 23:53:28.769883 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.769861    2551 scope.go:117] "RemoveContainer" containerID="50216cbd38fc87689f3eea41f49c61f7c967b359d30777d6b80a0f7bc7096547"
Jul 31 23:53:28.770164 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:53:28.770144    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50216cbd38fc87689f3eea41f49c61f7c967b359d30777d6b80a0f7bc7096547\": container with ID starting with 50216cbd38fc87689f3eea41f49c61f7c967b359d30777d6b80a0f7bc7096547 not found: ID does not exist" containerID="50216cbd38fc87689f3eea41f49c61f7c967b359d30777d6b80a0f7bc7096547"
Jul 31 23:53:28.770215 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.770174    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50216cbd38fc87689f3eea41f49c61f7c967b359d30777d6b80a0f7bc7096547"} err="failed to get container status \"50216cbd38fc87689f3eea41f49c61f7c967b359d30777d6b80a0f7bc7096547\": rpc error: code = NotFound desc = could not find container \"50216cbd38fc87689f3eea41f49c61f7c967b359d30777d6b80a0f7bc7096547\": container with ID starting with 50216cbd38fc87689f3eea41f49c61f7c967b359d30777d6b80a0f7bc7096547 not found: ID does not exist"
Jul 31 23:53:28.781353 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.781331    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"]
Jul 31 23:53:28.789155 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:28.789133    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-qcvgm"]
Jul 31 23:53:29.946746 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:29.946710    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f8480a-023e-4860-b516-4be9f59e61b2" path="/var/lib/kubelet/pods/c4f8480a-023e-4860-b516-4be9f59e61b2/volumes"
Jul 31 23:53:33.926273 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:33.926239    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:34.203445 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:34.203361    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:36.573622 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.573593    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-wmrp8"]
Jul 31 23:53:36.573986 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.573911    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f8480a-023e-4860-b516-4be9f59e61b2" containerName="limitador"
Jul 31 23:53:36.573986 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.573944    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f8480a-023e-4860-b516-4be9f59e61b2" containerName="limitador"
Jul 31 23:53:36.574067 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.573995    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f8480a-023e-4860-b516-4be9f59e61b2" containerName="limitador"
Jul 31 23:53:36.578193 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.578177    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-wmrp8"
Jul 31 23:53:36.580258 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.580234    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xmj24\""
Jul 31 23:53:36.580904 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.580890    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 31 23:53:36.590479 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.590459    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-wmrp8"]
Jul 31 23:53:36.690221 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.690195    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49tlr\" (UniqueName: \"kubernetes.io/projected/fa7b0e9f-0ba9-45cd-a560-07e454750004-kube-api-access-49tlr\") pod \"authorino-7cf4765b85-wmrp8\" (UID: \"fa7b0e9f-0ba9-45cd-a560-07e454750004\") " pod="kuadrant-system/authorino-7cf4765b85-wmrp8"
Jul 31 23:53:36.690221 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.690224    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa7b0e9f-0ba9-45cd-a560-07e454750004-tls-cert\") pod \"authorino-7cf4765b85-wmrp8\" (UID: \"fa7b0e9f-0ba9-45cd-a560-07e454750004\") " pod="kuadrant-system/authorino-7cf4765b85-wmrp8"
Jul 31 23:53:36.791569 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.791539    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tlr\" (UniqueName: \"kubernetes.io/projected/fa7b0e9f-0ba9-45cd-a560-07e454750004-kube-api-access-49tlr\") pod \"authorino-7cf4765b85-wmrp8\" (UID: \"fa7b0e9f-0ba9-45cd-a560-07e454750004\") " pod="kuadrant-system/authorino-7cf4765b85-wmrp8"
Jul 31 23:53:36.791699 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.791576    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa7b0e9f-0ba9-45cd-a560-07e454750004-tls-cert\") pod \"authorino-7cf4765b85-wmrp8\" (UID: \"fa7b0e9f-0ba9-45cd-a560-07e454750004\") " pod="kuadrant-system/authorino-7cf4765b85-wmrp8"
Jul 31 23:53:36.794204 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.794179    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa7b0e9f-0ba9-45cd-a560-07e454750004-tls-cert\") pod \"authorino-7cf4765b85-wmrp8\" (UID: \"fa7b0e9f-0ba9-45cd-a560-07e454750004\") " pod="kuadrant-system/authorino-7cf4765b85-wmrp8"
Jul 31 23:53:36.802826 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.802806    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49tlr\" (UniqueName: \"kubernetes.io/projected/fa7b0e9f-0ba9-45cd-a560-07e454750004-kube-api-access-49tlr\") pod \"authorino-7cf4765b85-wmrp8\" (UID: \"fa7b0e9f-0ba9-45cd-a560-07e454750004\") " pod="kuadrant-system/authorino-7cf4765b85-wmrp8"
Jul 31 23:53:36.886433 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:36.886366    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-wmrp8"
Jul 31 23:53:37.012171 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:37.011932    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-wmrp8"]
Jul 31 23:53:37.014491 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:53:37.014461    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa7b0e9f_0ba9_45cd_a560_07e454750004.slice/crio-66a24bb3ee31464c520eadf9007a4bd45a035e85d908272087c91ab9f2bd9969 WatchSource:0}: Error finding container 66a24bb3ee31464c520eadf9007a4bd45a035e85d908272087c91ab9f2bd9969: Status 404 returned error can't find the container with id 66a24bb3ee31464c520eadf9007a4bd45a035e85d908272087c91ab9f2bd9969
Jul 31 23:53:37.801810 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:37.801776    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-wmrp8" event={"ID":"fa7b0e9f-0ba9-45cd-a560-07e454750004","Type":"ContainerStarted","Data":"66a24bb3ee31464c520eadf9007a4bd45a035e85d908272087c91ab9f2bd9969"}
Jul 31 23:53:38.807258 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:38.807225    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-wmrp8" event={"ID":"fa7b0e9f-0ba9-45cd-a560-07e454750004","Type":"ContainerStarted","Data":"2d22562495560ae8a32773c6867535b1af9183242c368633e381d1ef689a108f"}
Jul 31 23:53:38.824991 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:38.824615    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-wmrp8" podStartSLOduration=1.158073712 podStartE2EDuration="2.824590433s" totalImagesPullingTime="1.666516721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:53:38.821822876 +0000 UTC m=+461.515870288" watchObservedRunningTime="2026-07-31 23:53:38.824590433 +0000 UTC m=+461.518637932"
Jul 31 23:53:41.937882 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:41.937850    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:42.042361 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:53:42.042322    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:06.394706 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.394669    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:06.482784 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.482746    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:06.483320 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.483297    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-786bd6bfd6-spzff"]
Jul 31 23:54:06.487651 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.487631    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff"
Jul 31 23:54:06.489606 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.489585    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 31 23:54:06.489718 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.489651    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 23:54:06.489781 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.489761    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-7rh67\""
Jul 31 23:54:06.490306 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.490283    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 23:54:06.495022 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.495004    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-786bd6bfd6-spzff"]
Jul 31 23:54:06.549824 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.549792    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4gj2\" (UniqueName: \"kubernetes.io/projected/e0056bb9-f2c5-4115-abf3-6136f785c4a2-kube-api-access-d4gj2\") pod \"llmisvc-controller-manager-786bd6bfd6-spzff\" (UID: \"e0056bb9-f2c5-4115-abf3-6136f785c4a2\") " pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff"
Jul 31 23:54:06.549988 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.549830    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0056bb9-f2c5-4115-abf3-6136f785c4a2-cert\") pod \"llmisvc-controller-manager-786bd6bfd6-spzff\" (UID: \"e0056bb9-f2c5-4115-abf3-6136f785c4a2\") " pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff"
Jul 31 23:54:06.566428 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.566400    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:06.650488 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.650400    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gj2\" (UniqueName: \"kubernetes.io/projected/e0056bb9-f2c5-4115-abf3-6136f785c4a2-kube-api-access-d4gj2\") pod \"llmisvc-controller-manager-786bd6bfd6-spzff\" (UID: \"e0056bb9-f2c5-4115-abf3-6136f785c4a2\") " pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff"
Jul 31 23:54:06.650488 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.650445    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0056bb9-f2c5-4115-abf3-6136f785c4a2-cert\") pod \"llmisvc-controller-manager-786bd6bfd6-spzff\" (UID: \"e0056bb9-f2c5-4115-abf3-6136f785c4a2\") " pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff"
Jul 31 23:54:06.650661 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:54:06.650595    2551 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 31 23:54:06.650661 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:54:06.650659    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0056bb9-f2c5-4115-abf3-6136f785c4a2-cert podName:e0056bb9-f2c5-4115-abf3-6136f785c4a2 nodeName:}" failed. No retries permitted until 2026-07-31 23:54:07.1506418 +0000 UTC m=+489.844689210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e0056bb9-f2c5-4115-abf3-6136f785c4a2-cert") pod "llmisvc-controller-manager-786bd6bfd6-spzff" (UID: "e0056bb9-f2c5-4115-abf3-6136f785c4a2") : secret "llmisvc-webhook-server-cert" not found
Jul 31 23:54:06.658838 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.658806    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4gj2\" (UniqueName: \"kubernetes.io/projected/e0056bb9-f2c5-4115-abf3-6136f785c4a2-kube-api-access-d4gj2\") pod \"llmisvc-controller-manager-786bd6bfd6-spzff\" (UID: \"e0056bb9-f2c5-4115-abf3-6136f785c4a2\") " pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff"
Jul 31 23:54:06.732063 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.732030    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:06.964843 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:06.964759    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:07.061757 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:07.061723    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:07.154005 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:07.153972    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0056bb9-f2c5-4115-abf3-6136f785c4a2-cert\") pod \"llmisvc-controller-manager-786bd6bfd6-spzff\" (UID: \"e0056bb9-f2c5-4115-abf3-6136f785c4a2\") " pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff"
Jul 31 23:54:07.156309 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:07.156287    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0056bb9-f2c5-4115-abf3-6136f785c4a2-cert\") pod \"llmisvc-controller-manager-786bd6bfd6-spzff\" (UID: \"e0056bb9-f2c5-4115-abf3-6136f785c4a2\") " pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff"
Jul 31 23:54:07.162355 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:07.162321    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:07.263163 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:07.263127    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:07.398011 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:07.397977    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff"
Jul 31 23:54:07.524425 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:07.524231    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-786bd6bfd6-spzff"]
Jul 31 23:54:07.527081 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:54:07.527049    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode0056bb9_f2c5_4115_abf3_6136f785c4a2.slice/crio-0c48c5295679b4903724f8c5f4ca0455ab71d6bb948c921662f4c204a76cf957 WatchSource:0}: Error finding container 0c48c5295679b4903724f8c5f4ca0455ab71d6bb948c921662f4c204a76cf957: Status 404 returned error can't find the container with id 0c48c5295679b4903724f8c5f4ca0455ab71d6bb948c921662f4c204a76cf957
Jul 31 23:54:07.917184 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:07.917113    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff" event={"ID":"e0056bb9-f2c5-4115-abf3-6136f785c4a2","Type":"ContainerStarted","Data":"0c48c5295679b4903724f8c5f4ca0455ab71d6bb948c921662f4c204a76cf957"}
Jul 31 23:54:10.931992 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:10.931953    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff" event={"ID":"e0056bb9-f2c5-4115-abf3-6136f785c4a2","Type":"ContainerStarted","Data":"490ed601594647c527d6dee9c06bceb2070b75aff85719357d02f34d029ff652"}
Jul 31 23:54:10.932347 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:10.932031    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff"
Jul 31 23:54:10.947873 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:10.947715    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff" podStartSLOduration=2.209866399 podStartE2EDuration="4.94770068s" totalImagesPullingTime="2.737834281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:54:10.947250386 +0000 UTC m=+493.641297795" watchObservedRunningTime="2026-07-31 23:54:10.94770068 +0000 UTC m=+493.641748096"
Jul 31 23:54:15.985360 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:15.985327    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:16.039345 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:16.039316    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:41.938295 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:41.938262    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-786bd6bfd6-spzff"
Jul 31 23:54:43.921859 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:43.921814    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:43.985296 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:43.985225    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:52.593575 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:52.593500    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:54:52.693752 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:54:52.693718    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:22.810638 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:55:22.810603    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:22.875562 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:55:22.875528    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:22.941075 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:55:22.941041    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:23.433526 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:55:23.433490    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:23.533646 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:55:23.533613    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:23.589479 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:55:23.589440    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:35.073562 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:55:35.073524    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:35.174287 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:55:35.174253    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:43.953367 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:56:43.953322    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:56.039800 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:56:56.039765    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:56.139061 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:56:56.139024    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:05.916118 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:05.916084    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:07.834277 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:07.834221    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:07.935015 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:07.934976    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:14.086563 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.086529    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:14.362694 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.362620    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:14.497826 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.497792    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:14.572989 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.572950    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 31 23:57:14.576377 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.576356    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.578554 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.578530    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 23:57:14.578713 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.578695    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 31 23:57:14.578791 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.578732    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 31 23:57:14.579493 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.579468    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-zbgmz\""
Jul 31 23:57:14.579611 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.579498    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 31 23:57:14.579611 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.579471    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 31 23:57:14.579744 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.579639    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-sv6zn\""
Jul 31 23:57:14.580421 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.580400    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6llpv\" (UniqueName: \"kubernetes.io/projected/d70c6565-40db-4c56-a61c-000df6c442ca-kube-api-access-6llpv\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.580477 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.580436    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70c6565-40db-4c56-a61c-000df6c442ca-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.580477 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.580458    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.580562 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.580479    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.580596 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.580563    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.580634 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.580616    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d70c6565-40db-4c56-a61c-000df6c442ca-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.580712 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.580667    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.580712 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.580703    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.584806 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.584786    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 31 23:57:14.681627 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.681556    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.681627 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.681592    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d70c6565-40db-4c56-a61c-000df6c442ca-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.681791 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.681626    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.681791 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.681654    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.681791 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.681715    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6llpv\" (UniqueName: \"kubernetes.io/projected/d70c6565-40db-4c56-a61c-000df6c442ca-kube-api-access-6llpv\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.681791 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.681743    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70c6565-40db-4c56-a61c-000df6c442ca-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.681791 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.681776    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.682067 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.681805    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.682067 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.681969    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.682067 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.682008    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.682224 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.682113    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.682224 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.682178    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.682297 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.682273    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d70c6565-40db-4c56-a61c-000df6c442ca-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.684002 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.683981    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.684158 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.684138    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70c6565-40db-4c56-a61c-000df6c442ca-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.697610 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.697582    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6llpv\" (UniqueName: \"kubernetes.io/projected/d70c6565-40db-4c56-a61c-000df6c442ca-kube-api-access-6llpv\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:14.736686 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.736658    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:14.888751 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:14.888724    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 23:57:15.022995 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:15.022968    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 31 23:57:15.025292 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:57:15.025269    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70c6565_40db_4c56_a61c_000df6c442ca.slice/crio-d10e19c00851604fa92028a3a4c70f88f8345733b6d37be26452d571fcde680e WatchSource:0}: Error finding container d10e19c00851604fa92028a3a4c70f88f8345733b6d37be26452d571fcde680e: Status 404 returned error can't find the container with id d10e19c00851604fa92028a3a4c70f88f8345733b6d37be26452d571fcde680e
Jul 31 23:57:15.027409 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:15.027392    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:57:15.644888 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:15.644779    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d70c6565-40db-4c56-a61c-000df6c442ca","Type":"ContainerStarted","Data":"d10e19c00851604fa92028a3a4c70f88f8345733b6d37be26452d571fcde680e"}
Jul 31 23:57:19.664702 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:19.664667    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d70c6565-40db-4c56-a61c-000df6c442ca","Type":"ContainerStarted","Data":"7da128d074d7ea6cc69749774926a88772558e7b0a8b8fa86fd63d7082361e0e"}
Jul 31 23:57:22.680017 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:22.679979    2551 generic.go:356] "Generic (PLEG): container finished" podID="d70c6565-40db-4c56-a61c-000df6c442ca" containerID="7da128d074d7ea6cc69749774926a88772558e7b0a8b8fa86fd63d7082361e0e" exitCode=0
Jul 31 23:57:22.680388 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:22.680071    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d70c6565-40db-4c56-a61c-000df6c442ca","Type":"ContainerDied","Data":"7da128d074d7ea6cc69749774926a88772558e7b0a8b8fa86fd63d7082361e0e"}
Jul 31 23:57:25.769788 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:25.769663    2551 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": read tcp 10.133.0.2:59078->10.133.0.19:15021: read: connection reset by peer" start-of-body=
Jul 31 23:57:25.769788 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:25.769736    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7" podUID="cbeff71f-8a02-4ed2-94cd-aa1ced7671ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": read tcp 10.133.0.2:59078->10.133.0.19:15021: read: connection reset by peer"
Jul 31 23:57:26.708036 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:26.707992    2551 generic.go:356] "Generic (PLEG): container finished" podID="cbeff71f-8a02-4ed2-94cd-aa1ced7671ac" containerID="f89eb622de866391c1722b4a6c3d16dba205da322f033094a99a0e1593dbf68a" exitCode=0
Jul 31 23:57:26.708223 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:26.708072    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7" event={"ID":"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac","Type":"ContainerDied","Data":"f89eb622de866391c1722b4a6c3d16dba205da322f033094a99a0e1593dbf68a"}
Jul 31 23:57:26.708604 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:26.708578    2551 scope.go:117] "RemoveContainer" containerID="f89eb622de866391c1722b4a6c3d16dba205da322f033094a99a0e1593dbf68a"
Jul 31 23:57:26.709878 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:26.709766    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 23:57:26.709878 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:26.709860    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 23:57:26.710073 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:26.709908    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 23:57:27.731637 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:27.731557    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7" event={"ID":"cbeff71f-8a02-4ed2-94cd-aa1ced7671ac","Type":"ContainerStarted","Data":"96438ce3781d585ac39d6eef870ddd23d148d20531f4b28c99c80ca41c7cc0c2"}
Jul 31 23:57:28.718749 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:28.718303    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:57:28.718749 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:28.718429    2551 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Jul 31 23:57:28.718749 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:28.718474    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7" podUID="cbeff71f-8a02-4ed2-94cd-aa1ced7671ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Jul 31 23:57:29.721136 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:29.719792    2551 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Jul 31 23:57:29.721136 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:29.719856    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7" podUID="cbeff71f-8a02-4ed2-94cd-aa1ced7671ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Jul 31 23:57:30.123211 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.123068    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:30.307952 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.298894    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"]
Jul 31 23:57:30.330676 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.328714    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"
Jul 31 23:57:30.333340 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.331940    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:30.333340 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.332330    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-ff85b\""
Jul 31 23:57:30.333340 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.332562    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 31 23:57:30.333340 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.332756    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 31 23:57:30.333340 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.332969    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-xr99c\""
Jul 31 23:57:30.333340 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.333033    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"]
Jul 31 23:57:30.333340 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.333172    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 31 23:57:30.365286 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.365259    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27pjj\" (UniqueName: \"kubernetes.io/projected/7c6647f6-419b-4c1c-95ce-0f5725efb169-kube-api-access-27pjj\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k\" (UID: \"7c6647f6-419b-4c1c-95ce-0f5725efb169\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"
Jul 31 23:57:30.365447 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.365402    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6647f6-419b-4c1c-95ce-0f5725efb169-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k\" (UID: \"7c6647f6-419b-4c1c-95ce-0f5725efb169\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"
Jul 31 23:57:30.466550 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.466472    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27pjj\" (UniqueName: \"kubernetes.io/projected/7c6647f6-419b-4c1c-95ce-0f5725efb169-kube-api-access-27pjj\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k\" (UID: \"7c6647f6-419b-4c1c-95ce-0f5725efb169\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"
Jul 31 23:57:30.466703 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.466581    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6647f6-419b-4c1c-95ce-0f5725efb169-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k\" (UID: \"7c6647f6-419b-4c1c-95ce-0f5725efb169\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"
Jul 31 23:57:30.469868 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.469817    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6647f6-419b-4c1c-95ce-0f5725efb169-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k\" (UID: \"7c6647f6-419b-4c1c-95ce-0f5725efb169\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"
Jul 31 23:57:30.474342 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.474296    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27pjj\" (UniqueName: \"kubernetes.io/projected/7c6647f6-419b-4c1c-95ce-0f5725efb169-kube-api-access-27pjj\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k\" (UID: \"7c6647f6-419b-4c1c-95ce-0f5725efb169\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"
Jul 31 23:57:30.643550 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.643411    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"
Jul 31 23:57:30.725158 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.724614    2551 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Jul 31 23:57:30.725158 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:30.724741    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7" podUID="cbeff71f-8a02-4ed2-94cd-aa1ced7671ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Jul 31 23:57:31.721943 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:31.721089    2551 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Jul 31 23:57:31.721943 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:31.721166    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7" podUID="cbeff71f-8a02-4ed2-94cd-aa1ced7671ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Jul 31 23:57:32.717538 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:32.717414    2551 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Jul 31 23:57:32.717538 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:32.717488    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7" podUID="cbeff71f-8a02-4ed2-94cd-aa1ced7671ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Jul 31 23:57:33.722030 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:33.721997    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:57:33.722665 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:33.722600    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:57:33.723647 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:33.723625    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7"
Jul 31 23:57:37.605754 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:37.605673    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"]
Jul 31 23:57:37.791329 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:37.791234    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k" event={"ID":"7c6647f6-419b-4c1c-95ce-0f5725efb169","Type":"ContainerStarted","Data":"2efc570fe1930387c56ec0273ab6083ddc69fe0c7975c7b5625fa931b680062b"}
Jul 31 23:57:45.436393 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:45.436338    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:54.885613 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:54.885567    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d70c6565-40db-4c56-a61c-000df6c442ca","Type":"ContainerStarted","Data":"846a430510942271e99970cf17b4bdcc6d82bee42197374e75aa5e5e7b4b7b01"}
Jul 31 23:57:54.887185 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:54.887157    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k" event={"ID":"7c6647f6-419b-4c1c-95ce-0f5725efb169","Type":"ContainerStarted","Data":"ec6934e0989b07efbca810b8d2cc5d95abd441f03abb1bb627f864e1394b929c"}
Jul 31 23:57:54.887308 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:54.887211    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"
Jul 31 23:57:54.922101 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:54.922044    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.65528804 podStartE2EDuration="40.92202458s" totalImagesPullingTime="35.881309531s" totalInitContainerRuntime="2.385427009s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:57:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:57:54.905205564 +0000 UTC m=+717.599252994" watchObservedRunningTime="2026-07-31 23:57:54.92202458 +0000 UTC m=+717.616071997"
Jul 31 23:57:54.922365 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:54.922334    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k" podStartSLOduration=8.549871394 podStartE2EDuration="24.922323625s" totalImagesPullingTime="16.372452231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:57:54.921143047 +0000 UTC m=+717.615190464" watchObservedRunningTime="2026-07-31 23:57:54.922323625 +0000 UTC m=+717.616371042"
Jul 31 23:57:59.658060 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:57:59.658022    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:25.894313 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:25.894281    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"
Jul 31 23:58:32.291106 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:32.291068    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"]
Jul 31 23:58:32.304414 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:32.304346    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:32.405138 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:32.405098    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:35.894195 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:35.894157    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k" podUID="7c6647f6-419b-4c1c-95ce-0f5725efb169" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 23:58:45.893470 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:45.893427    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k" podUID="7c6647f6-419b-4c1c-95ce-0f5725efb169" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 23:58:47.314857 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:47.314813    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k" podUID="7c6647f6-419b-4c1c-95ce-0f5725efb169" containerName="main" containerID="cri-o://ec6934e0989b07efbca810b8d2cc5d95abd441f03abb1bb627f864e1394b929c" gracePeriod=45
Jul 31 23:58:47.577970 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:47.577893    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"
Jul 31 23:58:47.688173 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:47.688134    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6647f6-419b-4c1c-95ce-0f5725efb169-tls-certs\") pod \"7c6647f6-419b-4c1c-95ce-0f5725efb169\" (UID: \"7c6647f6-419b-4c1c-95ce-0f5725efb169\") "
Jul 31 23:58:47.688358 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:47.688185    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27pjj\" (UniqueName: \"kubernetes.io/projected/7c6647f6-419b-4c1c-95ce-0f5725efb169-kube-api-access-27pjj\") pod \"7c6647f6-419b-4c1c-95ce-0f5725efb169\" (UID: \"7c6647f6-419b-4c1c-95ce-0f5725efb169\") "
Jul 31 23:58:47.690376 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:47.690346    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6647f6-419b-4c1c-95ce-0f5725efb169-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c6647f6-419b-4c1c-95ce-0f5725efb169" (UID: "7c6647f6-419b-4c1c-95ce-0f5725efb169"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:58:47.690518 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:47.690382    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6647f6-419b-4c1c-95ce-0f5725efb169-kube-api-access-27pjj" (OuterVolumeSpecName: "kube-api-access-27pjj") pod "7c6647f6-419b-4c1c-95ce-0f5725efb169" (UID: "7c6647f6-419b-4c1c-95ce-0f5725efb169"). InnerVolumeSpecName "kube-api-access-27pjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:58:47.789468 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:47.789423    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6647f6-419b-4c1c-95ce-0f5725efb169-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:58:47.789468 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:47.789452    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27pjj\" (UniqueName: \"kubernetes.io/projected/7c6647f6-419b-4c1c-95ce-0f5725efb169-kube-api-access-27pjj\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Jul 31 23:58:48.121004 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:48.120969    2551 generic.go:356] "Generic (PLEG): container finished" podID="7c6647f6-419b-4c1c-95ce-0f5725efb169" containerID="ec6934e0989b07efbca810b8d2cc5d95abd441f03abb1bb627f864e1394b929c" exitCode=0
Jul 31 23:58:48.121188 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:48.121036    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"
Jul 31 23:58:48.121188 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:48.121055    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k" event={"ID":"7c6647f6-419b-4c1c-95ce-0f5725efb169","Type":"ContainerDied","Data":"ec6934e0989b07efbca810b8d2cc5d95abd441f03abb1bb627f864e1394b929c"}
Jul 31 23:58:48.121188 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:48.121115    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k" event={"ID":"7c6647f6-419b-4c1c-95ce-0f5725efb169","Type":"ContainerDied","Data":"2efc570fe1930387c56ec0273ab6083ddc69fe0c7975c7b5625fa931b680062b"}
Jul 31 23:58:48.121188 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:48.121140    2551 scope.go:117] "RemoveContainer" containerID="ec6934e0989b07efbca810b8d2cc5d95abd441f03abb1bb627f864e1394b929c"
Jul 31 23:58:48.129446 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:48.129426    2551 scope.go:117] "RemoveContainer" containerID="ec6934e0989b07efbca810b8d2cc5d95abd441f03abb1bb627f864e1394b929c"
Jul 31 23:58:48.129708 ip-10-0-131-136 kubenswrapper[2551]: E0731 23:58:48.129683    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6934e0989b07efbca810b8d2cc5d95abd441f03abb1bb627f864e1394b929c\": container with ID starting with ec6934e0989b07efbca810b8d2cc5d95abd441f03abb1bb627f864e1394b929c not found: ID does not exist" containerID="ec6934e0989b07efbca810b8d2cc5d95abd441f03abb1bb627f864e1394b929c"
Jul 31 23:58:48.129764 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:48.129708    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6934e0989b07efbca810b8d2cc5d95abd441f03abb1bb627f864e1394b929c"} err="failed to get container status \"ec6934e0989b07efbca810b8d2cc5d95abd441f03abb1bb627f864e1394b929c\": rpc error: code = NotFound desc = could not find container \"ec6934e0989b07efbca810b8d2cc5d95abd441f03abb1bb627f864e1394b929c\": container with ID starting with ec6934e0989b07efbca810b8d2cc5d95abd441f03abb1bb627f864e1394b929c not found: ID does not exist"
Jul 31 23:58:48.136349 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:48.136326    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"]
Jul 31 23:58:48.141687 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:48.141664    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-n5z4k"]
Jul 31 23:58:49.946617 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:49.946585    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c6647f6-419b-4c1c-95ce-0f5725efb169" path="/var/lib/kubelet/pods/7c6647f6-419b-4c1c-95ce-0f5725efb169/volumes"
Jul 31 23:58:52.342433 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:52.342397    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:52.438296 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:52.438263    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:58.790220 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:58:58.790175    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:04.399572 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:04.399534    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:04.476449 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:04.476417    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:10.229908 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.229844    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:10.457449 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.457419    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"]
Jul 31 23:59:10.457768 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.457757    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6647f6-419b-4c1c-95ce-0f5725efb169" containerName="main"
Jul 31 23:59:10.457812 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.457776    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6647f6-419b-4c1c-95ce-0f5725efb169" containerName="main"
Jul 31 23:59:10.457845 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.457829    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6647f6-419b-4c1c-95ce-0f5725efb169" containerName="main"
Jul 31 23:59:10.462233 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.462215    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"
Jul 31 23:59:10.464377 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.464351    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 31 23:59:10.464377 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.464351    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-7j7k6\""
Jul 31 23:59:10.467171 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.465611    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-75l5n\""
Jul 31 23:59:10.467171 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.466219    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 31 23:59:10.471951 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.467491    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 31 23:59:10.474972 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.474912    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"]
Jul 31 23:59:10.485012 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.484947    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:10.598159 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.598118    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wz7j\" (UniqueName: \"kubernetes.io/projected/ca800633-bc91-45e9-b285-fe6c3708ca17-kube-api-access-9wz7j\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg\" (UID: \"ca800633-bc91-45e9-b285-fe6c3708ca17\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"
Jul 31 23:59:10.598343 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.598176    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca800633-bc91-45e9-b285-fe6c3708ca17-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg\" (UID: \"ca800633-bc91-45e9-b285-fe6c3708ca17\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"
Jul 31 23:59:10.699461 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.699422    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wz7j\" (UniqueName: \"kubernetes.io/projected/ca800633-bc91-45e9-b285-fe6c3708ca17-kube-api-access-9wz7j\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg\" (UID: \"ca800633-bc91-45e9-b285-fe6c3708ca17\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"
Jul 31 23:59:10.699668 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.699489    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca800633-bc91-45e9-b285-fe6c3708ca17-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg\" (UID: \"ca800633-bc91-45e9-b285-fe6c3708ca17\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"
Jul 31 23:59:10.702107 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.702074    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca800633-bc91-45e9-b285-fe6c3708ca17-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg\" (UID: \"ca800633-bc91-45e9-b285-fe6c3708ca17\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"
Jul 31 23:59:10.708016 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.707991    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wz7j\" (UniqueName: \"kubernetes.io/projected/ca800633-bc91-45e9-b285-fe6c3708ca17-kube-api-access-9wz7j\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg\" (UID: \"ca800633-bc91-45e9-b285-fe6c3708ca17\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"
Jul 31 23:59:10.777663 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.777557    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"
Jul 31 23:59:10.919018 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:10.918984    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"]
Jul 31 23:59:10.921021 ip-10-0-131-136 kubenswrapper[2551]: W0731 23:59:10.920984    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca800633_bc91_45e9_b285_fe6c3708ca17.slice/crio-6ed914e0b20aee2981952750921d9fd5a4648af262f7af53e926e72f16659ff6 WatchSource:0}: Error finding container 6ed914e0b20aee2981952750921d9fd5a4648af262f7af53e926e72f16659ff6: Status 404 returned error can't find the container with id 6ed914e0b20aee2981952750921d9fd5a4648af262f7af53e926e72f16659ff6
Jul 31 23:59:11.222616 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:11.222577    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg" event={"ID":"ca800633-bc91-45e9-b285-fe6c3708ca17","Type":"ContainerStarted","Data":"bdcda0a39c383bad1ff6766fdd24b007a8ee874803ecb99ed81424b34d3ad0d6"}
Jul 31 23:59:11.222779 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:11.222639    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg" event={"ID":"ca800633-bc91-45e9-b285-fe6c3708ca17","Type":"ContainerStarted","Data":"6ed914e0b20aee2981952750921d9fd5a4648af262f7af53e926e72f16659ff6"}
Jul 31 23:59:11.222779 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:11.222725    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"
Jul 31 23:59:11.261935 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:11.261862    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg" podStartSLOduration=1.261839212 podStartE2EDuration="1.261839212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:59:11.259607574 +0000 UTC m=+793.953654988" watchObservedRunningTime="2026-07-31 23:59:11.261839212 +0000 UTC m=+793.955886632"
Jul 31 23:59:19.203355 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:19.203318    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:30.150953 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:30.150860    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:30.212532 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:30.212495    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:42.227835 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:42.227801    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"
Jul 31 23:59:46.529224 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:46.529190    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 31 23:59:48.526741 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:48.526702    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:48.530308 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:48.530284    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"]
Jul 31 23:59:48.590312 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:48.590281    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:52.228772 ip-10-0-131-136 kubenswrapper[2551]: I0731 23:59:52.228731    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg" podUID="ca800633-bc91-45e9-b285-fe6c3708ca17" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:00:01.551572 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:01.551513    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="d70c6565-40db-4c56-a61c-000df6c442ca" containerName="main" containerID="cri-o://846a430510942271e99970cf17b4bdcc6d82bee42197374e75aa5e5e7b4b7b01" gracePeriod=45
Aug 01 00:00:02.228724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.228685    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg" podUID="ca800633-bc91-45e9-b285-fe6c3708ca17" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:00:02.314379 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.314345    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 01 00:00:02.439933 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.439897    2551 generic.go:356] "Generic (PLEG): container finished" podID="d70c6565-40db-4c56-a61c-000df6c442ca" containerID="846a430510942271e99970cf17b4bdcc6d82bee42197374e75aa5e5e7b4b7b01" exitCode=0
Aug 01 00:00:02.440131 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.439960    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 01 00:00:02.440131 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.439962    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d70c6565-40db-4c56-a61c-000df6c442ca","Type":"ContainerDied","Data":"846a430510942271e99970cf17b4bdcc6d82bee42197374e75aa5e5e7b4b7b01"}
Aug 01 00:00:02.440131 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.440103    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d70c6565-40db-4c56-a61c-000df6c442ca","Type":"ContainerDied","Data":"d10e19c00851604fa92028a3a4c70f88f8345733b6d37be26452d571fcde680e"}
Aug 01 00:00:02.440303 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.440140    2551 scope.go:117] "RemoveContainer" containerID="846a430510942271e99970cf17b4bdcc6d82bee42197374e75aa5e5e7b4b7b01"
Aug 01 00:00:02.448810 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.448785    2551 scope.go:117] "RemoveContainer" containerID="7da128d074d7ea6cc69749774926a88772558e7b0a8b8fa86fd63d7082361e0e"
Aug 01 00:00:02.456196 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.456167    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-dshm\") pod \"d70c6565-40db-4c56-a61c-000df6c442ca\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") "
Aug 01 00:00:02.456324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.456203    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-home\") pod \"d70c6565-40db-4c56-a61c-000df6c442ca\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") "
Aug 01 00:00:02.456324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.456236    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-kserve-provision-location\") pod \"d70c6565-40db-4c56-a61c-000df6c442ca\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") "
Aug 01 00:00:02.456324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.456286    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-model-cache\") pod \"d70c6565-40db-4c56-a61c-000df6c442ca\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") "
Aug 01 00:00:02.456324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.456320    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-tmp-dir\") pod \"d70c6565-40db-4c56-a61c-000df6c442ca\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") "
Aug 01 00:00:02.456577 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.456345    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d70c6565-40db-4c56-a61c-000df6c442ca-cabundle-cert\") pod \"d70c6565-40db-4c56-a61c-000df6c442ca\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") "
Aug 01 00:00:02.456577 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.456386    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6llpv\" (UniqueName: \"kubernetes.io/projected/d70c6565-40db-4c56-a61c-000df6c442ca-kube-api-access-6llpv\") pod \"d70c6565-40db-4c56-a61c-000df6c442ca\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") "
Aug 01 00:00:02.456690 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.456672    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70c6565-40db-4c56-a61c-000df6c442ca-tls-certs\") pod \"d70c6565-40db-4c56-a61c-000df6c442ca\" (UID: \"d70c6565-40db-4c56-a61c-000df6c442ca\") "
Aug 01 00:00:02.456896 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.456867    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70c6565-40db-4c56-a61c-000df6c442ca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d70c6565-40db-4c56-a61c-000df6c442ca" (UID: "d70c6565-40db-4c56-a61c-000df6c442ca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:02.456896 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.456889    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-home" (OuterVolumeSpecName: "home") pod "d70c6565-40db-4c56-a61c-000df6c442ca" (UID: "d70c6565-40db-4c56-a61c-000df6c442ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:02.457078 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.457043    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d70c6565-40db-4c56-a61c-000df6c442ca-cabundle-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.457078 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.457065    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.457271 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.457200    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "d70c6565-40db-4c56-a61c-000df6c442ca" (UID: "d70c6565-40db-4c56-a61c-000df6c442ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:02.457375 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.457354    2551 scope.go:117] "RemoveContainer" containerID="846a430510942271e99970cf17b4bdcc6d82bee42197374e75aa5e5e7b4b7b01"
Aug 01 00:00:02.457963 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:00:02.457791    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846a430510942271e99970cf17b4bdcc6d82bee42197374e75aa5e5e7b4b7b01\": container with ID starting with 846a430510942271e99970cf17b4bdcc6d82bee42197374e75aa5e5e7b4b7b01 not found: ID does not exist" containerID="846a430510942271e99970cf17b4bdcc6d82bee42197374e75aa5e5e7b4b7b01"
Aug 01 00:00:02.457963 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.457830    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846a430510942271e99970cf17b4bdcc6d82bee42197374e75aa5e5e7b4b7b01"} err="failed to get container status \"846a430510942271e99970cf17b4bdcc6d82bee42197374e75aa5e5e7b4b7b01\": rpc error: code = NotFound desc = could not find container \"846a430510942271e99970cf17b4bdcc6d82bee42197374e75aa5e5e7b4b7b01\": container with ID starting with 846a430510942271e99970cf17b4bdcc6d82bee42197374e75aa5e5e7b4b7b01 not found: ID does not exist"
Aug 01 00:00:02.457963 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.457857    2551 scope.go:117] "RemoveContainer" containerID="7da128d074d7ea6cc69749774926a88772558e7b0a8b8fa86fd63d7082361e0e"
Aug 01 00:00:02.458339 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:00:02.458278    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da128d074d7ea6cc69749774926a88772558e7b0a8b8fa86fd63d7082361e0e\": container with ID starting with 7da128d074d7ea6cc69749774926a88772558e7b0a8b8fa86fd63d7082361e0e not found: ID does not exist" containerID="7da128d074d7ea6cc69749774926a88772558e7b0a8b8fa86fd63d7082361e0e"
Aug 01 00:00:02.458339 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.458314    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da128d074d7ea6cc69749774926a88772558e7b0a8b8fa86fd63d7082361e0e"} err="failed to get container status \"7da128d074d7ea6cc69749774926a88772558e7b0a8b8fa86fd63d7082361e0e\": rpc error: code = NotFound desc = could not find container \"7da128d074d7ea6cc69749774926a88772558e7b0a8b8fa86fd63d7082361e0e\": container with ID starting with 7da128d074d7ea6cc69749774926a88772558e7b0a8b8fa86fd63d7082361e0e not found: ID does not exist"
Aug 01 00:00:02.458960 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.458935    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-dshm" (OuterVolumeSpecName: "dshm") pod "d70c6565-40db-4c56-a61c-000df6c442ca" (UID: "d70c6565-40db-4c56-a61c-000df6c442ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:02.459254 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.459226    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70c6565-40db-4c56-a61c-000df6c442ca-kube-api-access-6llpv" (OuterVolumeSpecName: "kube-api-access-6llpv") pod "d70c6565-40db-4c56-a61c-000df6c442ca" (UID: "d70c6565-40db-4c56-a61c-000df6c442ca"). InnerVolumeSpecName "kube-api-access-6llpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:00:02.459320 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.459272    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70c6565-40db-4c56-a61c-000df6c442ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d70c6565-40db-4c56-a61c-000df6c442ca" (UID: "d70c6565-40db-4c56-a61c-000df6c442ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:00:02.468807 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.468772    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d70c6565-40db-4c56-a61c-000df6c442ca" (UID: "d70c6565-40db-4c56-a61c-000df6c442ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:02.477438 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.477387    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d70c6565-40db-4c56-a61c-000df6c442ca" (UID: "d70c6565-40db-4c56-a61c-000df6c442ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:02.557999 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.557958    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.557999 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.557992    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-kserve-provision-location\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.557999 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.558004    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.558435 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.558013    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70c6565-40db-4c56-a61c-000df6c442ca-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.558435 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.558023    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6llpv\" (UniqueName: \"kubernetes.io/projected/d70c6565-40db-4c56-a61c-000df6c442ca-kube-api-access-6llpv\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.558435 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.558032    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70c6565-40db-4c56-a61c-000df6c442ca-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.763149 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.763115    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Aug 01 00:00:02.766714 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:02.766688    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Aug 01 00:00:03.550195 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:03.550154    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg" podUID="ca800633-bc91-45e9-b285-fe6c3708ca17" containerName="main" containerID="cri-o://bdcda0a39c383bad1ff6766fdd24b007a8ee874803ecb99ed81424b34d3ad0d6" gracePeriod=45
Aug 01 00:00:03.792545 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:03.792519    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"
Aug 01 00:00:03.946943 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:03.946855    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d70c6565-40db-4c56-a61c-000df6c442ca" path="/var/lib/kubelet/pods/d70c6565-40db-4c56-a61c-000df6c442ca/volumes"
Aug 01 00:00:03.970974 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:03.970941    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca800633-bc91-45e9-b285-fe6c3708ca17-tls-certs\") pod \"ca800633-bc91-45e9-b285-fe6c3708ca17\" (UID: \"ca800633-bc91-45e9-b285-fe6c3708ca17\") "
Aug 01 00:00:03.971129 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:03.971032    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wz7j\" (UniqueName: \"kubernetes.io/projected/ca800633-bc91-45e9-b285-fe6c3708ca17-kube-api-access-9wz7j\") pod \"ca800633-bc91-45e9-b285-fe6c3708ca17\" (UID: \"ca800633-bc91-45e9-b285-fe6c3708ca17\") "
Aug 01 00:00:03.973216 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:03.973188    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca800633-bc91-45e9-b285-fe6c3708ca17-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca800633-bc91-45e9-b285-fe6c3708ca17" (UID: "ca800633-bc91-45e9-b285-fe6c3708ca17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:00:03.973352 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:03.973326    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca800633-bc91-45e9-b285-fe6c3708ca17-kube-api-access-9wz7j" (OuterVolumeSpecName: "kube-api-access-9wz7j") pod "ca800633-bc91-45e9-b285-fe6c3708ca17" (UID: "ca800633-bc91-45e9-b285-fe6c3708ca17"). InnerVolumeSpecName "kube-api-access-9wz7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:00:04.071975 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:04.071906    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wz7j\" (UniqueName: \"kubernetes.io/projected/ca800633-bc91-45e9-b285-fe6c3708ca17-kube-api-access-9wz7j\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:04.071975 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:04.071964    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca800633-bc91-45e9-b285-fe6c3708ca17-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:04.450739 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:04.450704    2551 generic.go:356] "Generic (PLEG): container finished" podID="ca800633-bc91-45e9-b285-fe6c3708ca17" containerID="bdcda0a39c383bad1ff6766fdd24b007a8ee874803ecb99ed81424b34d3ad0d6" exitCode=0
Aug 01 00:00:04.450960 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:04.450750    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"
Aug 01 00:00:04.450960 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:04.450777    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg" event={"ID":"ca800633-bc91-45e9-b285-fe6c3708ca17","Type":"ContainerDied","Data":"bdcda0a39c383bad1ff6766fdd24b007a8ee874803ecb99ed81424b34d3ad0d6"}
Aug 01 00:00:04.450960 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:04.450831    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg" event={"ID":"ca800633-bc91-45e9-b285-fe6c3708ca17","Type":"ContainerDied","Data":"6ed914e0b20aee2981952750921d9fd5a4648af262f7af53e926e72f16659ff6"}
Aug 01 00:00:04.450960 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:04.450860    2551 scope.go:117] "RemoveContainer" containerID="bdcda0a39c383bad1ff6766fdd24b007a8ee874803ecb99ed81424b34d3ad0d6"
Aug 01 00:00:04.461675 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:04.461647    2551 scope.go:117] "RemoveContainer" containerID="bdcda0a39c383bad1ff6766fdd24b007a8ee874803ecb99ed81424b34d3ad0d6"
Aug 01 00:00:04.462313 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:00:04.462292    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdcda0a39c383bad1ff6766fdd24b007a8ee874803ecb99ed81424b34d3ad0d6\": container with ID starting with bdcda0a39c383bad1ff6766fdd24b007a8ee874803ecb99ed81424b34d3ad0d6 not found: ID does not exist" containerID="bdcda0a39c383bad1ff6766fdd24b007a8ee874803ecb99ed81424b34d3ad0d6"
Aug 01 00:00:04.462398 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:04.462326    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdcda0a39c383bad1ff6766fdd24b007a8ee874803ecb99ed81424b34d3ad0d6"} err="failed to get container status \"bdcda0a39c383bad1ff6766fdd24b007a8ee874803ecb99ed81424b34d3ad0d6\": rpc error: code = NotFound desc = could not find container \"bdcda0a39c383bad1ff6766fdd24b007a8ee874803ecb99ed81424b34d3ad0d6\": container with ID starting with bdcda0a39c383bad1ff6766fdd24b007a8ee874803ecb99ed81424b34d3ad0d6 not found: ID does not exist"
Aug 01 00:00:04.477205 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:04.477167    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"]
Aug 01 00:00:04.478653 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:04.478629    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-kp2xg"]
Aug 01 00:00:05.282869 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:05.282836    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:05.563394 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:05.563311    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:05.946416 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:05.946333    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca800633-bc91-45e9-b285-fe6c3708ca17" path="/var/lib/kubelet/pods/ca800633-bc91-45e9-b285-fe6c3708ca17/volumes"
Aug 01 00:00:08.494679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.494639    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:08.594379 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.594350    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:08.695648 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.695608    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"]
Aug 01 00:00:08.695966 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.695949    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d70c6565-40db-4c56-a61c-000df6c442ca" containerName="main"
Aug 01 00:00:08.695966 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.695966    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d70c6565-40db-4c56-a61c-000df6c442ca" containerName="main"
Aug 01 00:00:08.696122 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.695987    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca800633-bc91-45e9-b285-fe6c3708ca17" containerName="main"
Aug 01 00:00:08.696122 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.695993    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca800633-bc91-45e9-b285-fe6c3708ca17" containerName="main"
Aug 01 00:00:08.696122 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.696002    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d70c6565-40db-4c56-a61c-000df6c442ca" containerName="storage-initializer"
Aug 01 00:00:08.696122 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.696009    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d70c6565-40db-4c56-a61c-000df6c442ca" containerName="storage-initializer"
Aug 01 00:00:08.696122 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.696068    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d70c6565-40db-4c56-a61c-000df6c442ca" containerName="main"
Aug 01 00:00:08.696122 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.696080    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca800633-bc91-45e9-b285-fe6c3708ca17" containerName="main"
Aug 01 00:00:08.700419 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.700393    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.702633 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.702613    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-p2lzc\""
Aug 01 00:00:08.702881 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.702863    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Aug 01 00:00:08.702991 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.702942    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Aug 01 00:00:08.702991 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.702984    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Aug 01 00:00:08.711807 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.711786    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"]
Aug 01 00:00:08.736671 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.736624    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:08.816151 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.816072    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.816151 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.816119    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.816314 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.816161    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.816314 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.816193    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmq8f\" (UniqueName: \"kubernetes.io/projected/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-kube-api-access-nmq8f\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.816314 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.816235    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.816314 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.816281    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.816314 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.816309    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.816497 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.816366    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.816497 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.816423    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.845638 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.845609    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:08.917870 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.917828    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.917870 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.917873    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmq8f\" (UniqueName: \"kubernetes.io/projected/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-kube-api-access-nmq8f\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.918068 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.917938    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.918068 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.917981    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.918068 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.918001    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.918202 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.918178    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.918272 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.918246    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.918375 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.918278    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.918375 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.918303    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.918482 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.918441    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.918533 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.918497    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.918698 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.918677    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.918761 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.918702    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.918761 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.918750    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.920376 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.920352    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.920772 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.920749    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.934487 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.934464    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:08.934665 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:08.934644    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmq8f\" (UniqueName: \"kubernetes.io/projected/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-kube-api-access-nmq8f\") pod \"router-gateway-1-openshift-default-5b4d9458bd-sh2dx\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:09.011753 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:09.011717    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:09.151021 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:09.150913    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"]
Aug 01 00:00:09.153104 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:00:09.153077    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2fab56_57b5_45fc_85c1_ac340f92fb6c.slice/crio-d8a91ed7614e17771e528a56006af56743984d98306eb8e1dc703b5e292f85ce WatchSource:0}: Error finding container d8a91ed7614e17771e528a56006af56743984d98306eb8e1dc703b5e292f85ce: Status 404 returned error can't find the container with id d8a91ed7614e17771e528a56006af56743984d98306eb8e1dc703b5e292f85ce
Aug 01 00:00:09.155123 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:09.155092    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:00:09.155195 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:09.155158    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:00:09.155195 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:09.155185    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:00:09.473398 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:09.473362    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx" event={"ID":"dd2fab56-57b5-45fc-85c1-ac340f92fb6c","Type":"ContainerStarted","Data":"5e874976c12ff0883a7851926485bc840ad6f866746dad2664ada2baa662b2a6"}
Aug 01 00:00:09.473564 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:09.473430    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx" event={"ID":"dd2fab56-57b5-45fc-85c1-ac340f92fb6c","Type":"ContainerStarted","Data":"d8a91ed7614e17771e528a56006af56743984d98306eb8e1dc703b5e292f85ce"}
Aug 01 00:00:09.494388 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:09.494337    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx" podStartSLOduration=1.494322379 podStartE2EDuration="1.494322379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:00:09.492483381 +0000 UTC m=+852.186530798" watchObservedRunningTime="2026-08-01 00:00:09.494322379 +0000 UTC m=+852.188369795"
Aug 01 00:00:10.011886 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:10.011852    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:10.016779 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:10.016746    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:10.477907 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:10.477876    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:10.478888 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:10.478869    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:11.435837 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:11.435798    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:13.047057 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:13.047018    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:14.330057 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:14.330021    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:16.460281 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.460245    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"]
Aug 01 00:00:16.463479 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.463463    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.466420 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.466397    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-qh4kd\""
Aug 01 00:00:16.466420 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.466417    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Aug 01 00:00:16.466614 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.466422    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Aug 01 00:00:16.466614 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.466474    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Aug 01 00:00:16.472803 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.472783    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"]
Aug 01 00:00:16.484170 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.484135    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.484307 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.484191    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgpqd\" (UniqueName: \"kubernetes.io/projected/9fba60db-21d8-46bd-83a3-2bd82054cc51-kube-api-access-dgpqd\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.484307 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.484224    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fba60db-21d8-46bd-83a3-2bd82054cc51-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.484307 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.484280    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.484477 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.484319    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.484477 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.484380    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.547824 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.547794    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:16.585157 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.585116    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.585354 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.585166    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.585354 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.585206    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.585354 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.585278    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.585354 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.585322    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgpqd\" (UniqueName: \"kubernetes.io/projected/9fba60db-21d8-46bd-83a3-2bd82054cc51-kube-api-access-dgpqd\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.585576 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.585371    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fba60db-21d8-46bd-83a3-2bd82054cc51-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.585576 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.585511    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.585576 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.585557    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.585738 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.585658    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.587489 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.587468    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.587835 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.587814    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fba60db-21d8-46bd-83a3-2bd82054cc51-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.592793 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.592774    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgpqd\" (UniqueName: \"kubernetes.io/projected/9fba60db-21d8-46bd-83a3-2bd82054cc51-kube-api-access-dgpqd\") pod \"scheduler-inline-config-test-kserve-8655bc757-6fglc\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.709466 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.709437    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"]
Aug 01 00:00:16.713075 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.713018    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"
Aug 01 00:00:16.715069 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.715049    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-hllg2\""
Aug 01 00:00:16.720835 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.720815    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"]
Aug 01 00:00:16.730708 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.730677    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:16.774155 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.774130    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:16.787200 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.787173    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdqcc\" (UniqueName: \"kubernetes.io/projected/994e3772-2dc1-44ad-9f92-8f21f4ba8ca4-kube-api-access-xdqcc\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv\" (UID: \"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"
Aug 01 00:00:16.787300 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.787278    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994e3772-2dc1-44ad-9f92-8f21f4ba8ca4-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv\" (UID: \"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"
Aug 01 00:00:16.888594 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.888563    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994e3772-2dc1-44ad-9f92-8f21f4ba8ca4-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv\" (UID: \"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"
Aug 01 00:00:16.888780 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.888692    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdqcc\" (UniqueName: \"kubernetes.io/projected/994e3772-2dc1-44ad-9f92-8f21f4ba8ca4-kube-api-access-xdqcc\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv\" (UID: \"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"
Aug 01 00:00:16.891155 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.891130    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994e3772-2dc1-44ad-9f92-8f21f4ba8ca4-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv\" (UID: \"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"
Aug 01 00:00:16.899754 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.899729    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdqcc\" (UniqueName: \"kubernetes.io/projected/994e3772-2dc1-44ad-9f92-8f21f4ba8ca4-kube-api-access-xdqcc\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv\" (UID: \"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"
Aug 01 00:00:16.905748 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:16.905728    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"]
Aug 01 00:00:16.908158 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:00:16.908133    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fba60db_21d8_46bd_83a3_2bd82054cc51.slice/crio-a3941df0d38119d3165425ff1f2e5f1b93ffbe9ff3f5d6d9804d925d7fd9934b WatchSource:0}: Error finding container a3941df0d38119d3165425ff1f2e5f1b93ffbe9ff3f5d6d9804d925d7fd9934b: Status 404 returned error can't find the container with id a3941df0d38119d3165425ff1f2e5f1b93ffbe9ff3f5d6d9804d925d7fd9934b
Aug 01 00:00:17.023852 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:17.023818    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"
Aug 01 00:00:17.171328 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:17.168853    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"]
Aug 01 00:00:17.172908 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:00:17.172878    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994e3772_2dc1_44ad_9f92_8f21f4ba8ca4.slice/crio-5d22d4f56aaee52419162e204f2eeea0605e4af6d6fb7c68b565a35779f48b1d WatchSource:0}: Error finding container 5d22d4f56aaee52419162e204f2eeea0605e4af6d6fb7c68b565a35779f48b1d: Status 404 returned error can't find the container with id 5d22d4f56aaee52419162e204f2eeea0605e4af6d6fb7c68b565a35779f48b1d
Aug 01 00:00:17.518047 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:17.518013    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv" event={"ID":"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4","Type":"ContainerStarted","Data":"3c7587b4d71df6dc393a1e6ef57a55496c2bb9fb923cef56e09bfa27210b782a"}
Aug 01 00:00:17.518511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:17.518074    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv" event={"ID":"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4","Type":"ContainerStarted","Data":"5d22d4f56aaee52419162e204f2eeea0605e4af6d6fb7c68b565a35779f48b1d"}
Aug 01 00:00:17.518511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:17.518160    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"
Aug 01 00:00:17.519570 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:17.519540    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc" event={"ID":"9fba60db-21d8-46bd-83a3-2bd82054cc51","Type":"ContainerStarted","Data":"a3941df0d38119d3165425ff1f2e5f1b93ffbe9ff3f5d6d9804d925d7fd9934b"}
Aug 01 00:00:17.533445 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:17.533389    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv" podStartSLOduration=1.533370551 podStartE2EDuration="1.533370551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:00:17.53271699 +0000 UTC m=+860.226764408" watchObservedRunningTime="2026-08-01 00:00:17.533370551 +0000 UTC m=+860.227417964"
Aug 01 00:00:18.111276 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:18.111236    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:18.212593 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:18.212542    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:20.224202 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:20.224164    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:20.324632 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:20.324598    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:24.989434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:24.989398    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:27.303911 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:27.303836    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:27.404531 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:27.404399    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:27.507749 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:27.507714    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:27.717428 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:27.717355    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:27.813189 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:27.813150    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:31.059978 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:31.059943    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:32.293583 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:32.293546    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"]
Aug 01 00:00:32.294073 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:32.293797    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx" podUID="dd2fab56-57b5-45fc-85c1-ac340f92fb6c" containerName="istio-proxy" containerID="cri-o://5e874976c12ff0883a7851926485bc840ad6f866746dad2664ada2baa662b2a6" gracePeriod=30
Aug 01 00:00:32.383285 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:32.383252    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:35.138957 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.138931    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:35.252199 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252169    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-credential-socket\") pod \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") "
Aug 01 00:00:35.252349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252209    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-podinfo\") pod \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") "
Aug 01 00:00:35.252349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252225    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-workload-certs\") pod \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") "
Aug 01 00:00:35.252349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252259    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-workload-socket\") pod \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") "
Aug 01 00:00:35.252349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252281    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istiod-ca-cert\") pod \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") "
Aug 01 00:00:35.252349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252321    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-token\") pod \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") "
Aug 01 00:00:35.252349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252337    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmq8f\" (UniqueName: \"kubernetes.io/projected/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-kube-api-access-nmq8f\") pod \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") "
Aug 01 00:00:35.252666 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252380    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-envoy\") pod \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") "
Aug 01 00:00:35.252666 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252417    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dd2fab56-57b5-45fc-85c1-ac340f92fb6c" (UID: "dd2fab56-57b5-45fc-85c1-ac340f92fb6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:35.252666 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252428    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-data\") pod \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\" (UID: \"dd2fab56-57b5-45fc-85c1-ac340f92fb6c\") "
Aug 01 00:00:35.252666 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252630    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dd2fab56-57b5-45fc-85c1-ac340f92fb6c" (UID: "dd2fab56-57b5-45fc-85c1-ac340f92fb6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:35.252863 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252749    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dd2fab56-57b5-45fc-85c1-ac340f92fb6c" (UID: "dd2fab56-57b5-45fc-85c1-ac340f92fb6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:35.252863 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252774    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:35.252863 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252826    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:35.252863 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252831    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dd2fab56-57b5-45fc-85c1-ac340f92fb6c" (UID: "dd2fab56-57b5-45fc-85c1-ac340f92fb6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:35.253050 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.252963    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-data" (OuterVolumeSpecName: "istio-data") pod "dd2fab56-57b5-45fc-85c1-ac340f92fb6c" (UID: "dd2fab56-57b5-45fc-85c1-ac340f92fb6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:35.254502 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.254477    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dd2fab56-57b5-45fc-85c1-ac340f92fb6c" (UID: "dd2fab56-57b5-45fc-85c1-ac340f92fb6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 00:00:35.254642 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.254599    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-token" (OuterVolumeSpecName: "istio-token") pod "dd2fab56-57b5-45fc-85c1-ac340f92fb6c" (UID: "dd2fab56-57b5-45fc-85c1-ac340f92fb6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:00:35.254892 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.254872    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dd2fab56-57b5-45fc-85c1-ac340f92fb6c" (UID: "dd2fab56-57b5-45fc-85c1-ac340f92fb6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:35.254971 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.254876    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-kube-api-access-nmq8f" (OuterVolumeSpecName: "kube-api-access-nmq8f") pod "dd2fab56-57b5-45fc-85c1-ac340f92fb6c" (UID: "dd2fab56-57b5-45fc-85c1-ac340f92fb6c"). InnerVolumeSpecName "kube-api-access-nmq8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:00:35.353582 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.353560    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:35.353582 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.353582    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:35.353582 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.353591    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:35.353774 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.353601    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:35.353774 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.353609    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:35.353774 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.353617    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmq8f\" (UniqueName: \"kubernetes.io/projected/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-kube-api-access-nmq8f\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:35.353774 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.353625    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2fab56-57b5-45fc-85c1-ac340f92fb6c-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:00:35.601869 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.601826    2551 generic.go:356] "Generic (PLEG): container finished" podID="dd2fab56-57b5-45fc-85c1-ac340f92fb6c" containerID="5e874976c12ff0883a7851926485bc840ad6f866746dad2664ada2baa662b2a6" exitCode=0
Aug 01 00:00:35.602082 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.601875    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"
Aug 01 00:00:35.602082 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.601894    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx" event={"ID":"dd2fab56-57b5-45fc-85c1-ac340f92fb6c","Type":"ContainerDied","Data":"5e874976c12ff0883a7851926485bc840ad6f866746dad2664ada2baa662b2a6"}
Aug 01 00:00:35.602082 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.601993    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx" event={"ID":"dd2fab56-57b5-45fc-85c1-ac340f92fb6c","Type":"ContainerDied","Data":"d8a91ed7614e17771e528a56006af56743984d98306eb8e1dc703b5e292f85ce"}
Aug 01 00:00:35.602082 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.602021    2551 scope.go:117] "RemoveContainer" containerID="5e874976c12ff0883a7851926485bc840ad6f866746dad2664ada2baa662b2a6"
Aug 01 00:00:35.610570 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.610554    2551 scope.go:117] "RemoveContainer" containerID="5e874976c12ff0883a7851926485bc840ad6f866746dad2664ada2baa662b2a6"
Aug 01 00:00:35.610835 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:00:35.610810    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e874976c12ff0883a7851926485bc840ad6f866746dad2664ada2baa662b2a6\": container with ID starting with 5e874976c12ff0883a7851926485bc840ad6f866746dad2664ada2baa662b2a6 not found: ID does not exist" containerID="5e874976c12ff0883a7851926485bc840ad6f866746dad2664ada2baa662b2a6"
Aug 01 00:00:35.610895 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.610847    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e874976c12ff0883a7851926485bc840ad6f866746dad2664ada2baa662b2a6"} err="failed to get container status \"5e874976c12ff0883a7851926485bc840ad6f866746dad2664ada2baa662b2a6\": rpc error: code = NotFound desc = could not find container \"5e874976c12ff0883a7851926485bc840ad6f866746dad2664ada2baa662b2a6\": container with ID starting with 5e874976c12ff0883a7851926485bc840ad6f866746dad2664ada2baa662b2a6 not found: ID does not exist"
Aug 01 00:00:35.622602 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.622570    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"]
Aug 01 00:00:35.625807 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.625788    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-sh2dx"]
Aug 01 00:00:35.946931 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:35.946845    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd2fab56-57b5-45fc-85c1-ac340f92fb6c" path="/var/lib/kubelet/pods/dd2fab56-57b5-45fc-85c1-ac340f92fb6c/volumes"
Aug 01 00:00:40.628362 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:40.628321    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc" event={"ID":"9fba60db-21d8-46bd-83a3-2bd82054cc51","Type":"ContainerStarted","Data":"23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62"}
Aug 01 00:00:40.646995 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:40.646908    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc" podStartSLOduration=1.745332742 podStartE2EDuration="24.646887836s" totalImagesPullingTime="22.901555094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:00:40.645500384 +0000 UTC m=+883.339547800" watchObservedRunningTime="2026-08-01 00:00:40.646887836 +0000 UTC m=+883.340935254"
Aug 01 00:00:40.774632 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:40.774584    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:41.705890 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:41.705855    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:43.687157 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:43.687125    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:46.774974 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:46.774932    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:46.786999 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:46.786975    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:47.672680 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:47.672649    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:00:48.527208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:48.527178    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"
Aug 01 00:00:53.465296 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:53.465217    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:56.190844 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:56.190816    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:56.359200 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:56.359167    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:56.460298 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:00:56.460216    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:01.270899 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:01.270860    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:06.420117 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:06.420080    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:06.723674 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:06.723641    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:08.071620 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:08.071586    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:09.329265 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:09.329232    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:09.429743 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:09.429660    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:12.702218 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:12.702184    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:13.807636 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:13.807597    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:13.918065 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:13.918029    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:18.300598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:18.300559    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:18.496038 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:18.495996    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:19.897132 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:19.897101    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:20.771678 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:20.771638    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"]
Aug 01 00:01:20.777907 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:20.777868    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:20.779493 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:20.779466    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"]
Aug 01 00:01:20.788274 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:01:20.788243    2551 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:01:20.788274 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:01:20.788274 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:01:20.788274 ip-10-0-131-136 kubenswrapper[2551]:  > containerID="23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62" cmd=["/bin/sleep","15"]
Aug 01 00:01:20.788568 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:20.788293    2551 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc" message="" err=<
Aug 01 00:01:20.788568 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:01:20.788568 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:01:20.788568 ip-10-0-131-136 kubenswrapper[2551]:  >
Aug 01 00:01:20.788568 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:01:20.788306    2551 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:01:20.788568 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:01:20.788568 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:01:20.788568 ip-10-0-131-136 kubenswrapper[2551]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc" podUID="9fba60db-21d8-46bd-83a3-2bd82054cc51" containerName="main" containerID="cri-o://23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62"
Aug 01 00:01:20.788568 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:20.788336    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc" podUID="9fba60db-21d8-46bd-83a3-2bd82054cc51" containerName="main" containerID="cri-o://23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62" gracePeriod=60
Aug 01 00:01:20.845267 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:20.845237    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:21.137620 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.137595    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:01:21.242226 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.242193    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-model-cache\") pod \"9fba60db-21d8-46bd-83a3-2bd82054cc51\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") "
Aug 01 00:01:21.242226 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.242231    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-tmp-dir\") pod \"9fba60db-21d8-46bd-83a3-2bd82054cc51\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") "
Aug 01 00:01:21.242468 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.242263    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-home\") pod \"9fba60db-21d8-46bd-83a3-2bd82054cc51\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") "
Aug 01 00:01:21.242468 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.242297    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fba60db-21d8-46bd-83a3-2bd82054cc51-tls-certs\") pod \"9fba60db-21d8-46bd-83a3-2bd82054cc51\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") "
Aug 01 00:01:21.242468 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.242318    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgpqd\" (UniqueName: \"kubernetes.io/projected/9fba60db-21d8-46bd-83a3-2bd82054cc51-kube-api-access-dgpqd\") pod \"9fba60db-21d8-46bd-83a3-2bd82054cc51\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") "
Aug 01 00:01:21.242468 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.242373    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-dshm\") pod \"9fba60db-21d8-46bd-83a3-2bd82054cc51\" (UID: \"9fba60db-21d8-46bd-83a3-2bd82054cc51\") "
Aug 01 00:01:21.242677 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.242510    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9fba60db-21d8-46bd-83a3-2bd82054cc51" (UID: "9fba60db-21d8-46bd-83a3-2bd82054cc51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:21.242677 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.242533    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-model-cache" (OuterVolumeSpecName: "model-cache") pod "9fba60db-21d8-46bd-83a3-2bd82054cc51" (UID: "9fba60db-21d8-46bd-83a3-2bd82054cc51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:21.242677 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.242575    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-home" (OuterVolumeSpecName: "home") pod "9fba60db-21d8-46bd-83a3-2bd82054cc51" (UID: "9fba60db-21d8-46bd-83a3-2bd82054cc51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:21.242835 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.242762    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:01:21.242835 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.242809    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:01:21.242949 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.242824    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:01:21.244718 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.244683    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-dshm" (OuterVolumeSpecName: "dshm") pod "9fba60db-21d8-46bd-83a3-2bd82054cc51" (UID: "9fba60db-21d8-46bd-83a3-2bd82054cc51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:21.244842 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.244731    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fba60db-21d8-46bd-83a3-2bd82054cc51-kube-api-access-dgpqd" (OuterVolumeSpecName: "kube-api-access-dgpqd") pod "9fba60db-21d8-46bd-83a3-2bd82054cc51" (UID: "9fba60db-21d8-46bd-83a3-2bd82054cc51"). InnerVolumeSpecName "kube-api-access-dgpqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:01:21.244842 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.244748    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fba60db-21d8-46bd-83a3-2bd82054cc51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9fba60db-21d8-46bd-83a3-2bd82054cc51" (UID: "9fba60db-21d8-46bd-83a3-2bd82054cc51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:01:21.343410 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.343346    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fba60db-21d8-46bd-83a3-2bd82054cc51-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:01:21.343410 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.343370    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fba60db-21d8-46bd-83a3-2bd82054cc51-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:01:21.343410 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.343381    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgpqd\" (UniqueName: \"kubernetes.io/projected/9fba60db-21d8-46bd-83a3-2bd82054cc51-kube-api-access-dgpqd\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:01:21.810160 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.810127    2551 generic.go:356] "Generic (PLEG): container finished" podID="9fba60db-21d8-46bd-83a3-2bd82054cc51" containerID="23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62" exitCode=0
Aug 01 00:01:21.810316 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.810184    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"
Aug 01 00:01:21.810316 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.810194    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc" event={"ID":"9fba60db-21d8-46bd-83a3-2bd82054cc51","Type":"ContainerDied","Data":"23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62"}
Aug 01 00:01:21.810316 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.810238    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc" event={"ID":"9fba60db-21d8-46bd-83a3-2bd82054cc51","Type":"ContainerDied","Data":"a3941df0d38119d3165425ff1f2e5f1b93ffbe9ff3f5d6d9804d925d7fd9934b"}
Aug 01 00:01:21.810316 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.810261    2551 scope.go:117] "RemoveContainer" containerID="23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62"
Aug 01 00:01:21.818733 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.818717    2551 scope.go:117] "RemoveContainer" containerID="23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62"
Aug 01 00:01:21.819023 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:01:21.819002    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62\": container with ID starting with 23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62 not found: ID does not exist" containerID="23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62"
Aug 01 00:01:21.819100 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.819030    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62"} err="failed to get container status \"23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62\": rpc error: code = NotFound desc = could not find container \"23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62\": container with ID starting with 23a030f510794f237b120ab41ddbd45e34be8d69ae2e9d3a8bee43014a2ccf62 not found: ID does not exist"
Aug 01 00:01:21.828968 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.828944    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"]
Aug 01 00:01:21.832534 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.832515    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6fglc"]
Aug 01 00:01:21.947080 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:21.947055    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fba60db-21d8-46bd-83a3-2bd82054cc51" path="/var/lib/kubelet/pods/9fba60db-21d8-46bd-83a3-2bd82054cc51/volumes"
Aug 01 00:01:23.373051 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:23.372992    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:24.016788 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:24.016752    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:24.081399 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:24.081356    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:28.526856 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:28.526805    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv" podUID="994e3772-2dc1-44ad-9f92-8f21f4ba8ca4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:01:35.800176 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:35.800107    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv" podUID="994e3772-2dc1-44ad-9f92-8f21f4ba8ca4" containerName="main" containerID="cri-o://3c7587b4d71df6dc393a1e6ef57a55496c2bb9fb923cef56e09bfa27210b782a" gracePeriod=45
Aug 01 00:01:36.039565 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.039535    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"
Aug 01 00:01:36.172091 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.171995    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994e3772-2dc1-44ad-9f92-8f21f4ba8ca4-tls-certs\") pod \"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4\" (UID: \"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4\") "
Aug 01 00:01:36.172091 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.172052    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdqcc\" (UniqueName: \"kubernetes.io/projected/994e3772-2dc1-44ad-9f92-8f21f4ba8ca4-kube-api-access-xdqcc\") pod \"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4\" (UID: \"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4\") "
Aug 01 00:01:36.173785 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.173751    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994e3772-2dc1-44ad-9f92-8f21f4ba8ca4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "994e3772-2dc1-44ad-9f92-8f21f4ba8ca4" (UID: "994e3772-2dc1-44ad-9f92-8f21f4ba8ca4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:01:36.173984 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.173872    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994e3772-2dc1-44ad-9f92-8f21f4ba8ca4-kube-api-access-xdqcc" (OuterVolumeSpecName: "kube-api-access-xdqcc") pod "994e3772-2dc1-44ad-9f92-8f21f4ba8ca4" (UID: "994e3772-2dc1-44ad-9f92-8f21f4ba8ca4"). InnerVolumeSpecName "kube-api-access-xdqcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:01:36.273124 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.273084    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994e3772-2dc1-44ad-9f92-8f21f4ba8ca4-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:01:36.273124 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.273117    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdqcc\" (UniqueName: \"kubernetes.io/projected/994e3772-2dc1-44ad-9f92-8f21f4ba8ca4-kube-api-access-xdqcc\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:01:36.870200 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.870168    2551 generic.go:356] "Generic (PLEG): container finished" podID="994e3772-2dc1-44ad-9f92-8f21f4ba8ca4" containerID="3c7587b4d71df6dc393a1e6ef57a55496c2bb9fb923cef56e09bfa27210b782a" exitCode=0
Aug 01 00:01:36.870657 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.870211    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"
Aug 01 00:01:36.870657 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.870257    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv" event={"ID":"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4","Type":"ContainerDied","Data":"3c7587b4d71df6dc393a1e6ef57a55496c2bb9fb923cef56e09bfa27210b782a"}
Aug 01 00:01:36.870657 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.870299    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv" event={"ID":"994e3772-2dc1-44ad-9f92-8f21f4ba8ca4","Type":"ContainerDied","Data":"5d22d4f56aaee52419162e204f2eeea0605e4af6d6fb7c68b565a35779f48b1d"}
Aug 01 00:01:36.870657 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.870322    2551 scope.go:117] "RemoveContainer" containerID="3c7587b4d71df6dc393a1e6ef57a55496c2bb9fb923cef56e09bfa27210b782a"
Aug 01 00:01:36.878884 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.878866    2551 scope.go:117] "RemoveContainer" containerID="3c7587b4d71df6dc393a1e6ef57a55496c2bb9fb923cef56e09bfa27210b782a"
Aug 01 00:01:36.879165 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:01:36.879146    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7587b4d71df6dc393a1e6ef57a55496c2bb9fb923cef56e09bfa27210b782a\": container with ID starting with 3c7587b4d71df6dc393a1e6ef57a55496c2bb9fb923cef56e09bfa27210b782a not found: ID does not exist" containerID="3c7587b4d71df6dc393a1e6ef57a55496c2bb9fb923cef56e09bfa27210b782a"
Aug 01 00:01:36.879234 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.879172    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7587b4d71df6dc393a1e6ef57a55496c2bb9fb923cef56e09bfa27210b782a"} err="failed to get container status \"3c7587b4d71df6dc393a1e6ef57a55496c2bb9fb923cef56e09bfa27210b782a\": rpc error: code = NotFound desc = could not find container \"3c7587b4d71df6dc393a1e6ef57a55496c2bb9fb923cef56e09bfa27210b782a\": container with ID starting with 3c7587b4d71df6dc393a1e6ef57a55496c2bb9fb923cef56e09bfa27210b782a not found: ID does not exist"
Aug 01 00:01:36.890270 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.890249    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"]
Aug 01 00:01:36.893759 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:36.893738    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd524rfv"]
Aug 01 00:01:37.946198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:37.946168    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="994e3772-2dc1-44ad-9f92-8f21f4ba8ca4" path="/var/lib/kubelet/pods/994e3772-2dc1-44ad-9f92-8f21f4ba8ca4/volumes"
Aug 01 00:01:40.527740 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:40.527700    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:40.621036 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:40.620993    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:45.568002 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.567969    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"]
Aug 01 00:01:45.568376 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.568266    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="994e3772-2dc1-44ad-9f92-8f21f4ba8ca4" containerName="main"
Aug 01 00:01:45.568376 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.568277    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="994e3772-2dc1-44ad-9f92-8f21f4ba8ca4" containerName="main"
Aug 01 00:01:45.568376 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.568289    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fba60db-21d8-46bd-83a3-2bd82054cc51" containerName="main"
Aug 01 00:01:45.568376 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.568294    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fba60db-21d8-46bd-83a3-2bd82054cc51" containerName="main"
Aug 01 00:01:45.568376 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.568301    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd2fab56-57b5-45fc-85c1-ac340f92fb6c" containerName="istio-proxy"
Aug 01 00:01:45.568376 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.568307    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd2fab56-57b5-45fc-85c1-ac340f92fb6c" containerName="istio-proxy"
Aug 01 00:01:45.568376 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.568358    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd2fab56-57b5-45fc-85c1-ac340f92fb6c" containerName="istio-proxy"
Aug 01 00:01:45.568376 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.568366    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="994e3772-2dc1-44ad-9f92-8f21f4ba8ca4" containerName="main"
Aug 01 00:01:45.568376 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.568373    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fba60db-21d8-46bd-83a3-2bd82054cc51" containerName="main"
Aug 01 00:01:45.572694 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.572674    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.574710 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.574689    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Aug 01 00:01:45.575381 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.575356    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-57fjf\""
Aug 01 00:01:45.575474 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.575360    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Aug 01 00:01:45.575535 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.575483    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Aug 01 00:01:45.578936 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.578901    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"]
Aug 01 00:01:45.594491 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.594462    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:45.752151 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.752115    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.752326 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.752156    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.752326 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.752183    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.752326 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.752257    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9j5c\" (UniqueName: \"kubernetes.io/projected/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-kube-api-access-n9j5c\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.752326 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.752320    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.752463 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.752390    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.795672 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.795639    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"]
Aug 01 00:01:45.798822 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.798806    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"
Aug 01 00:01:45.800861 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.800839    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-rmflp\""
Aug 01 00:01:45.807132 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.807111    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"]
Aug 01 00:01:45.852955 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.852869    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.852955 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.852906    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.852955 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.852946    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.853208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.852965    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.853208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.852990    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9j5c\" (UniqueName: \"kubernetes.io/projected/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-kube-api-access-n9j5c\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.853328 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.853212    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.853328 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.853254    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.853328 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.853313    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.853440 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.853348    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.855283 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.855259    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.855450 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.855433    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.859911 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.859888    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9j5c\" (UniqueName: \"kubernetes.io/projected/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-kube-api-access-n9j5c\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.883835 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.883812    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:45.954340 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.954306    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2svdm\" (UniqueName: \"kubernetes.io/projected/582cd63e-5734-4cb7-a333-ef93384c59a4-kube-api-access-2svdm\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2\" (UID: \"582cd63e-5734-4cb7-a333-ef93384c59a4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"
Aug 01 00:01:45.954495 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:45.954422    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582cd63e-5734-4cb7-a333-ef93384c59a4-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2\" (UID: \"582cd63e-5734-4cb7-a333-ef93384c59a4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"
Aug 01 00:01:46.017712 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.017679    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"]
Aug 01 00:01:46.020321 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:01:46.020293    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fdaf72_6f59_49cf_b786_1e0e97ad4d87.slice/crio-b0c806923cf36d4c09468d67784f95384a8da82fe3504f5a6ce2fc767b2aabfc WatchSource:0}: Error finding container b0c806923cf36d4c09468d67784f95384a8da82fe3504f5a6ce2fc767b2aabfc: Status 404 returned error can't find the container with id b0c806923cf36d4c09468d67784f95384a8da82fe3504f5a6ce2fc767b2aabfc
Aug 01 00:01:46.055371 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.055344    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2svdm\" (UniqueName: \"kubernetes.io/projected/582cd63e-5734-4cb7-a333-ef93384c59a4-kube-api-access-2svdm\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2\" (UID: \"582cd63e-5734-4cb7-a333-ef93384c59a4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"
Aug 01 00:01:46.055458 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.055418    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582cd63e-5734-4cb7-a333-ef93384c59a4-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2\" (UID: \"582cd63e-5734-4cb7-a333-ef93384c59a4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"
Aug 01 00:01:46.057685 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.057665    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582cd63e-5734-4cb7-a333-ef93384c59a4-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2\" (UID: \"582cd63e-5734-4cb7-a333-ef93384c59a4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"
Aug 01 00:01:46.062732 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.062711    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2svdm\" (UniqueName: \"kubernetes.io/projected/582cd63e-5734-4cb7-a333-ef93384c59a4-kube-api-access-2svdm\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2\" (UID: \"582cd63e-5734-4cb7-a333-ef93384c59a4\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"
Aug 01 00:01:46.109415 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.109344    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"
Aug 01 00:01:46.237486 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.237465    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"]
Aug 01 00:01:46.239903 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:01:46.239876    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582cd63e_5734_4cb7_a333_ef93384c59a4.slice/crio-67f71f389d2c2161bfe4941274b54094b8357090be93d970245f12403112b5a9 WatchSource:0}: Error finding container 67f71f389d2c2161bfe4941274b54094b8357090be93d970245f12403112b5a9: Status 404 returned error can't find the container with id 67f71f389d2c2161bfe4941274b54094b8357090be93d970245f12403112b5a9
Aug 01 00:01:46.921669 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.921634    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2" event={"ID":"582cd63e-5734-4cb7-a333-ef93384c59a4","Type":"ContainerStarted","Data":"b298134c2b0ce9a12e9c809ae58fea7e1717292e76561d459556417ebc40c672"}
Aug 01 00:01:46.922141 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.921686    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2" event={"ID":"582cd63e-5734-4cb7-a333-ef93384c59a4","Type":"ContainerStarted","Data":"67f71f389d2c2161bfe4941274b54094b8357090be93d970245f12403112b5a9"}
Aug 01 00:01:46.922141 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.921714    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"
Aug 01 00:01:46.923140 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.923110    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx" event={"ID":"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87","Type":"ContainerStarted","Data":"fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818"}
Aug 01 00:01:46.923249 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.923167    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx" event={"ID":"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87","Type":"ContainerStarted","Data":"b0c806923cf36d4c09468d67784f95384a8da82fe3504f5a6ce2fc767b2aabfc"}
Aug 01 00:01:46.936497 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.936454    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2" podStartSLOduration=1.93643937 podStartE2EDuration="1.93643937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:01:46.934960556 +0000 UTC m=+949.629007963" watchObservedRunningTime="2026-08-01 00:01:46.93643937 +0000 UTC m=+949.630486785"
Aug 01 00:01:46.950560 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:46.950512    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx" podStartSLOduration=1.950497891 podStartE2EDuration="1.950497891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:01:46.950255451 +0000 UTC m=+949.644302868" watchObservedRunningTime="2026-08-01 00:01:46.950497891 +0000 UTC m=+949.644545308"
Aug 01 00:01:47.884305 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:47.884265    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:48.547316 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:48.547281    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:52.274175 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:52.274141    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:55.168130 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:55.168090    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:55.221190 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:55.221158    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:55.884233 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:55.884191    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:55.896770 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:55.896737    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:01:55.971710 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:01:55.971682    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:02:13.009782 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:13.009749    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:17.929890 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:17.929856    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"
Aug 01 00:02:19.030193 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.030161    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"]
Aug 01 00:02:19.032640 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.032605    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"]
Aug 01 00:02:19.040794 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:02:19.040758    2551 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:02:19.040794 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:02:19.040794 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:02:19.040794 ip-10-0-131-136 kubenswrapper[2551]:  > containerID="fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818" cmd=["/bin/sleep","15"]
Aug 01 00:02:19.041081 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.040810    2551 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx" message="" err=<
Aug 01 00:02:19.041081 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:02:19.041081 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:02:19.041081 ip-10-0-131-136 kubenswrapper[2551]:  >
Aug 01 00:02:19.041081 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:02:19.040825    2551 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:02:19.041081 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:02:19.041081 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:02:19.041081 ip-10-0-131-136 kubenswrapper[2551]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx" podUID="c7fdaf72-6f59-49cf-b786-1e0e97ad4d87" containerName="main" containerID="cri-o://fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818"
Aug 01 00:02:19.041081 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.040859    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx" podUID="c7fdaf72-6f59-49cf-b786-1e0e97ad4d87" containerName="main" containerID="cri-o://fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818" gracePeriod=60
Aug 01 00:02:19.043454 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.043428    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:19.096621 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.096587    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:19.779854 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.779826    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:02:19.833865 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.833839    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-home\") pod \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") "
Aug 01 00:02:19.834035 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.833888    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-tmp-dir\") pod \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") "
Aug 01 00:02:19.834035 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.833941    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-model-cache\") pod \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") "
Aug 01 00:02:19.834035 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.833999    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9j5c\" (UniqueName: \"kubernetes.io/projected/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-kube-api-access-n9j5c\") pod \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") "
Aug 01 00:02:19.834035 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.834035    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-tls-certs\") pod \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") "
Aug 01 00:02:19.834258 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.834057    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-dshm\") pod \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\" (UID: \"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87\") "
Aug 01 00:02:19.834258 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.834146    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-home" (OuterVolumeSpecName: "home") pod "c7fdaf72-6f59-49cf-b786-1e0e97ad4d87" (UID: "c7fdaf72-6f59-49cf-b786-1e0e97ad4d87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:02:19.834258 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.834235    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7fdaf72-6f59-49cf-b786-1e0e97ad4d87" (UID: "c7fdaf72-6f59-49cf-b786-1e0e97ad4d87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:02:19.834258 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.834245    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7fdaf72-6f59-49cf-b786-1e0e97ad4d87" (UID: "c7fdaf72-6f59-49cf-b786-1e0e97ad4d87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:02:19.834409 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.834368    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:02:19.834409 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.834388    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:02:19.834409 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.834402    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:02:19.836203 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.836175    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-dshm" (OuterVolumeSpecName: "dshm") pod "c7fdaf72-6f59-49cf-b786-1e0e97ad4d87" (UID: "c7fdaf72-6f59-49cf-b786-1e0e97ad4d87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:02:19.836297 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.836250    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7fdaf72-6f59-49cf-b786-1e0e97ad4d87" (UID: "c7fdaf72-6f59-49cf-b786-1e0e97ad4d87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:02:19.836297 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.836285    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-kube-api-access-n9j5c" (OuterVolumeSpecName: "kube-api-access-n9j5c") pod "c7fdaf72-6f59-49cf-b786-1e0e97ad4d87" (UID: "c7fdaf72-6f59-49cf-b786-1e0e97ad4d87"). InnerVolumeSpecName "kube-api-access-n9j5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:02:19.935673 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.935597    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9j5c\" (UniqueName: \"kubernetes.io/projected/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-kube-api-access-n9j5c\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:02:19.935673 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.935625    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:02:19.935673 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:19.935638    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:02:20.061847 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:20.061815    2551 generic.go:356] "Generic (PLEG): container finished" podID="c7fdaf72-6f59-49cf-b786-1e0e97ad4d87" containerID="fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818" exitCode=0
Aug 01 00:02:20.062319 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:20.061860    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"
Aug 01 00:02:20.062319 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:20.061898    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx" event={"ID":"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87","Type":"ContainerDied","Data":"fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818"}
Aug 01 00:02:20.062319 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:20.061985    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx" event={"ID":"c7fdaf72-6f59-49cf-b786-1e0e97ad4d87","Type":"ContainerDied","Data":"b0c806923cf36d4c09468d67784f95384a8da82fe3504f5a6ce2fc767b2aabfc"}
Aug 01 00:02:20.062319 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:20.062026    2551 scope.go:117] "RemoveContainer" containerID="fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818"
Aug 01 00:02:20.070370 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:20.070355    2551 scope.go:117] "RemoveContainer" containerID="fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818"
Aug 01 00:02:20.070615 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:02:20.070596    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818\": container with ID starting with fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818 not found: ID does not exist" containerID="fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818"
Aug 01 00:02:20.070662 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:20.070622    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818"} err="failed to get container status \"fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818\": rpc error: code = NotFound desc = could not find container \"fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818\": container with ID starting with fde9c36f5b78b9df4252ebf49afd7f4a7f3bc00ba327d4f158d205c5d652d818 not found: ID does not exist"
Aug 01 00:02:20.075966 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:20.075942    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"]
Aug 01 00:02:20.081543 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:20.081523    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9x9dx"]
Aug 01 00:02:21.946685 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:21.946653    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7fdaf72-6f59-49cf-b786-1e0e97ad4d87" path="/var/lib/kubelet/pods/c7fdaf72-6f59-49cf-b786-1e0e97ad4d87/volumes"
Aug 01 00:02:27.929178 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:27.929138    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2" podUID="582cd63e-5734-4cb7-a333-ef93384c59a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:02:34.052709 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:34.052647    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2" podUID="582cd63e-5734-4cb7-a333-ef93384c59a4" containerName="main" containerID="cri-o://b298134c2b0ce9a12e9c809ae58fea7e1717292e76561d459556417ebc40c672" gracePeriod=45
Aug 01 00:02:34.313416 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:34.313357    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"
Aug 01 00:02:34.450746 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:34.450708    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2svdm\" (UniqueName: \"kubernetes.io/projected/582cd63e-5734-4cb7-a333-ef93384c59a4-kube-api-access-2svdm\") pod \"582cd63e-5734-4cb7-a333-ef93384c59a4\" (UID: \"582cd63e-5734-4cb7-a333-ef93384c59a4\") "
Aug 01 00:02:34.450746 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:34.450750    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582cd63e-5734-4cb7-a333-ef93384c59a4-tls-certs\") pod \"582cd63e-5734-4cb7-a333-ef93384c59a4\" (UID: \"582cd63e-5734-4cb7-a333-ef93384c59a4\") "
Aug 01 00:02:34.452874 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:34.452842    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582cd63e-5734-4cb7-a333-ef93384c59a4-kube-api-access-2svdm" (OuterVolumeSpecName: "kube-api-access-2svdm") pod "582cd63e-5734-4cb7-a333-ef93384c59a4" (UID: "582cd63e-5734-4cb7-a333-ef93384c59a4"). InnerVolumeSpecName "kube-api-access-2svdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:02:34.452874 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:34.452843    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/582cd63e-5734-4cb7-a333-ef93384c59a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "582cd63e-5734-4cb7-a333-ef93384c59a4" (UID: "582cd63e-5734-4cb7-a333-ef93384c59a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:02:34.552035 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:34.552004    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2svdm\" (UniqueName: \"kubernetes.io/projected/582cd63e-5734-4cb7-a333-ef93384c59a4-kube-api-access-2svdm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:02:34.552035 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:34.552033    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582cd63e-5734-4cb7-a333-ef93384c59a4-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:02:35.122608 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:35.122574    2551 generic.go:356] "Generic (PLEG): container finished" podID="582cd63e-5734-4cb7-a333-ef93384c59a4" containerID="b298134c2b0ce9a12e9c809ae58fea7e1717292e76561d459556417ebc40c672" exitCode=0
Aug 01 00:02:35.122608 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:35.122612    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2" event={"ID":"582cd63e-5734-4cb7-a333-ef93384c59a4","Type":"ContainerDied","Data":"b298134c2b0ce9a12e9c809ae58fea7e1717292e76561d459556417ebc40c672"}
Aug 01 00:02:35.123124 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:35.122623    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"
Aug 01 00:02:35.123124 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:35.122652    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2" event={"ID":"582cd63e-5734-4cb7-a333-ef93384c59a4","Type":"ContainerDied","Data":"67f71f389d2c2161bfe4941274b54094b8357090be93d970245f12403112b5a9"}
Aug 01 00:02:35.123124 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:35.122673    2551 scope.go:117] "RemoveContainer" containerID="b298134c2b0ce9a12e9c809ae58fea7e1717292e76561d459556417ebc40c672"
Aug 01 00:02:35.137686 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:35.137666    2551 scope.go:117] "RemoveContainer" containerID="b298134c2b0ce9a12e9c809ae58fea7e1717292e76561d459556417ebc40c672"
Aug 01 00:02:35.137985 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:02:35.137954    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b298134c2b0ce9a12e9c809ae58fea7e1717292e76561d459556417ebc40c672\": container with ID starting with b298134c2b0ce9a12e9c809ae58fea7e1717292e76561d459556417ebc40c672 not found: ID does not exist" containerID="b298134c2b0ce9a12e9c809ae58fea7e1717292e76561d459556417ebc40c672"
Aug 01 00:02:35.138051 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:35.137993    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b298134c2b0ce9a12e9c809ae58fea7e1717292e76561d459556417ebc40c672"} err="failed to get container status \"b298134c2b0ce9a12e9c809ae58fea7e1717292e76561d459556417ebc40c672\": rpc error: code = NotFound desc = could not find container \"b298134c2b0ce9a12e9c809ae58fea7e1717292e76561d459556417ebc40c672\": container with ID starting with b298134c2b0ce9a12e9c809ae58fea7e1717292e76561d459556417ebc40c672 not found: ID does not exist"
Aug 01 00:02:35.146421 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:35.146397    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"]
Aug 01 00:02:35.151691 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:35.151670    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schewqth2"]
Aug 01 00:02:35.945959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:35.945910    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="582cd63e-5734-4cb7-a333-ef93384c59a4" path="/var/lib/kubelet/pods/582cd63e-5734-4cb7-a333-ef93384c59a4/volumes"
Aug 01 00:02:39.064457 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:39.064426    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:39.187738 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:39.187681    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:43.254552 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.254522    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"]
Aug 01 00:02:43.255155 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.254972    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7fdaf72-6f59-49cf-b786-1e0e97ad4d87" containerName="main"
Aug 01 00:02:43.255155 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.254991    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7fdaf72-6f59-49cf-b786-1e0e97ad4d87" containerName="main"
Aug 01 00:02:43.255155 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.255007    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582cd63e-5734-4cb7-a333-ef93384c59a4" containerName="main"
Aug 01 00:02:43.255155 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.255016    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="582cd63e-5734-4cb7-a333-ef93384c59a4" containerName="main"
Aug 01 00:02:43.255155 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.255106    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7fdaf72-6f59-49cf-b786-1e0e97ad4d87" containerName="main"
Aug 01 00:02:43.255155 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.255119    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="582cd63e-5734-4cb7-a333-ef93384c59a4" containerName="main"
Aug 01 00:02:43.259706 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.259681    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.261847 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.261824    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Aug 01 00:02:43.262522 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.262501    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Aug 01 00:02:43.262625 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.262518    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Aug 01 00:02:43.262823 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.262810    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-mwskk\""
Aug 01 00:02:43.266807 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.266784    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"]
Aug 01 00:02:43.291281 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.291255    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:43.335839 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.335811    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5j7k\" (UniqueName: \"kubernetes.io/projected/2b246367-cfbd-4a5e-9029-28b98dd6351f-kube-api-access-x5j7k\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.335994 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.335849    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.335994 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.335947    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.335994 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.335981    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.336183 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.336001    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b246367-cfbd-4a5e-9029-28b98dd6351f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.336183 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.336071    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.436670 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.436640    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.436838 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.436680    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5j7k\" (UniqueName: \"kubernetes.io/projected/2b246367-cfbd-4a5e-9029-28b98dd6351f-kube-api-access-x5j7k\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.436838 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.436707    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.436838 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.436756    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.436838 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.436779    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.436838 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.436800    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b246367-cfbd-4a5e-9029-28b98dd6351f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.437130 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.437105    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.437169 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.437138    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.437225 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.437209    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.439026 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.439006    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.439270 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.439254    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b246367-cfbd-4a5e-9029-28b98dd6351f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.445308 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.445284    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5j7k\" (UniqueName: \"kubernetes.io/projected/2b246367-cfbd-4a5e-9029-28b98dd6351f-kube-api-access-x5j7k\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.463952 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.463906    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"]
Aug 01 00:02:43.467344 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.467326    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"
Aug 01 00:02:43.469418 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.469396    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-b9mc4\""
Aug 01 00:02:43.475544 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.475523    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"]
Aug 01 00:02:43.481591 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.481572    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:43.537893 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.537827    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf8sw\" (UniqueName: \"kubernetes.io/projected/a381c347-f60d-410b-9727-ed23a4faa15b-kube-api-access-xf8sw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2\" (UID: \"a381c347-f60d-410b-9727-ed23a4faa15b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"
Aug 01 00:02:43.538010 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.537892    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a381c347-f60d-410b-9727-ed23a4faa15b-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2\" (UID: \"a381c347-f60d-410b-9727-ed23a4faa15b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"
Aug 01 00:02:43.571781 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.571757    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:43.638884 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.638848    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a381c347-f60d-410b-9727-ed23a4faa15b-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2\" (UID: \"a381c347-f60d-410b-9727-ed23a4faa15b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"
Aug 01 00:02:43.639150 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.638954    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf8sw\" (UniqueName: \"kubernetes.io/projected/a381c347-f60d-410b-9727-ed23a4faa15b-kube-api-access-xf8sw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2\" (UID: \"a381c347-f60d-410b-9727-ed23a4faa15b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"
Aug 01 00:02:43.641451 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.641421    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a381c347-f60d-410b-9727-ed23a4faa15b-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2\" (UID: \"a381c347-f60d-410b-9727-ed23a4faa15b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"
Aug 01 00:02:43.646954 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.646904    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf8sw\" (UniqueName: \"kubernetes.io/projected/a381c347-f60d-410b-9727-ed23a4faa15b-kube-api-access-xf8sw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2\" (UID: \"a381c347-f60d-410b-9727-ed23a4faa15b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"
Aug 01 00:02:43.699681 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.699654    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"]
Aug 01 00:02:43.702120 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:02:43.702096    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b246367_cfbd_4a5e_9029_28b98dd6351f.slice/crio-ae7353156a1eb987f38a94ca23295d9f2f1a098e58e69ce26d89b0451d27b419 WatchSource:0}: Error finding container ae7353156a1eb987f38a94ca23295d9f2f1a098e58e69ce26d89b0451d27b419: Status 404 returned error can't find the container with id ae7353156a1eb987f38a94ca23295d9f2f1a098e58e69ce26d89b0451d27b419
Aug 01 00:02:43.777971 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.777932    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"
Aug 01 00:02:43.903288 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:43.903263    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"]
Aug 01 00:02:43.905299 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:02:43.905275    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda381c347_f60d_410b_9727_ed23a4faa15b.slice/crio-76c481c86936a26fe10901c00696fe3af32ddcdc1d9c1c117bb5feaae2db8089 WatchSource:0}: Error finding container 76c481c86936a26fe10901c00696fe3af32ddcdc1d9c1c117bb5feaae2db8089: Status 404 returned error can't find the container with id 76c481c86936a26fe10901c00696fe3af32ddcdc1d9c1c117bb5feaae2db8089
Aug 01 00:02:44.161514 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:44.161419    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2" event={"ID":"a381c347-f60d-410b-9727-ed23a4faa15b","Type":"ContainerStarted","Data":"5ef5bea21f5b89ba40690a4d21f9fe989a0777fd4621b125b0bbd5086dd51304"}
Aug 01 00:02:44.161514 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:44.161483    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2" event={"ID":"a381c347-f60d-410b-9727-ed23a4faa15b","Type":"ContainerStarted","Data":"76c481c86936a26fe10901c00696fe3af32ddcdc1d9c1c117bb5feaae2db8089"}
Aug 01 00:02:44.161730 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:44.161540    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"
Aug 01 00:02:44.163035 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:44.163010    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm" event={"ID":"2b246367-cfbd-4a5e-9029-28b98dd6351f","Type":"ContainerStarted","Data":"02a54869f488d0c639c433a16efcd9713fe4724f9faf534842e553cb025b4894"}
Aug 01 00:02:44.163159 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:44.163054    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm" event={"ID":"2b246367-cfbd-4a5e-9029-28b98dd6351f","Type":"ContainerStarted","Data":"ae7353156a1eb987f38a94ca23295d9f2f1a098e58e69ce26d89b0451d27b419"}
Aug 01 00:02:44.184355 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:44.184306    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2" podStartSLOduration=1.184293012 podStartE2EDuration="1.184293012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:02:44.181941575 +0000 UTC m=+1006.875988981" watchObservedRunningTime="2026-08-01 00:02:44.184293012 +0000 UTC m=+1006.878340428"
Aug 01 00:02:44.572010 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:44.571977    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:45.809514 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:45.809477    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:46.059945 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:46.059851    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:51.484470 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:51.484430    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:51.570196 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:51.570166    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:53.572678 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:53.572646    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:53.585050 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:53.585026    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:53.602750 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:53.602702    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm" podStartSLOduration=10.602686322 podStartE2EDuration="10.602686322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:02:44.198600739 +0000 UTC m=+1006.892648158" watchObservedRunningTime="2026-08-01 00:02:53.602686322 +0000 UTC m=+1016.296733740"
Aug 01 00:02:54.215354 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:54.215318    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:02:59.026306 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:59.026260    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:59.077823 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:02:59.077793    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:01.987600 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:01.987567    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:15.169171 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:15.169136    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"
Aug 01 00:03:18.848173 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:18.848121    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:20.138589 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:20.138553    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:21.106672 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.106636    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"]
Aug 01 00:03:21.114111 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.114082    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"]
Aug 01 00:03:21.119649 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.119619    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:21.122063 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:03:21.122026    2551 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:03:21.122063 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:03:21.122063 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:03:21.122063 ip-10-0-131-136 kubenswrapper[2551]:  > containerID="02a54869f488d0c639c433a16efcd9713fe4724f9faf534842e553cb025b4894" cmd=["/bin/sleep","15"]
Aug 01 00:03:21.122349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.122084    2551 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm" message="" err=<
Aug 01 00:03:21.122349 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:03:21.122349 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:03:21.122349 ip-10-0-131-136 kubenswrapper[2551]:  >
Aug 01 00:03:21.122349 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:03:21.122099    2551 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:03:21.122349 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:03:21.122349 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:03:21.122349 ip-10-0-131-136 kubenswrapper[2551]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm" podUID="2b246367-cfbd-4a5e-9029-28b98dd6351f" containerName="main" containerID="cri-o://02a54869f488d0c639c433a16efcd9713fe4724f9faf534842e553cb025b4894"
Aug 01 00:03:21.122349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.122147    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm" podUID="2b246367-cfbd-4a5e-9029-28b98dd6351f" containerName="main" containerID="cri-o://02a54869f488d0c639c433a16efcd9713fe4724f9faf534842e553cb025b4894" gracePeriod=60
Aug 01 00:03:21.185471 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.185429    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:21.205499 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.205459    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm" podUID="2b246367-cfbd-4a5e-9029-28b98dd6351f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Aug 01 00:03:21.318324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.318262    2551 generic.go:356] "Generic (PLEG): container finished" podID="2b246367-cfbd-4a5e-9029-28b98dd6351f" containerID="02a54869f488d0c639c433a16efcd9713fe4724f9faf534842e553cb025b4894" exitCode=0
Aug 01 00:03:21.318483 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.318345    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm" event={"ID":"2b246367-cfbd-4a5e-9029-28b98dd6351f","Type":"ContainerDied","Data":"02a54869f488d0c639c433a16efcd9713fe4724f9faf534842e553cb025b4894"}
Aug 01 00:03:21.880365 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.880341    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:03:21.947434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.947408    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-model-cache\") pod \"2b246367-cfbd-4a5e-9029-28b98dd6351f\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") "
Aug 01 00:03:21.947588 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.947443    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-home\") pod \"2b246367-cfbd-4a5e-9029-28b98dd6351f\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") "
Aug 01 00:03:21.947588 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.947478    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5j7k\" (UniqueName: \"kubernetes.io/projected/2b246367-cfbd-4a5e-9029-28b98dd6351f-kube-api-access-x5j7k\") pod \"2b246367-cfbd-4a5e-9029-28b98dd6351f\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") "
Aug 01 00:03:21.947588 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.947504    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-tmp-dir\") pod \"2b246367-cfbd-4a5e-9029-28b98dd6351f\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") "
Aug 01 00:03:21.947588 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.947536    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b246367-cfbd-4a5e-9029-28b98dd6351f-tls-certs\") pod \"2b246367-cfbd-4a5e-9029-28b98dd6351f\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") "
Aug 01 00:03:21.947745 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.947610    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-dshm\") pod \"2b246367-cfbd-4a5e-9029-28b98dd6351f\" (UID: \"2b246367-cfbd-4a5e-9029-28b98dd6351f\") "
Aug 01 00:03:21.947745 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.947651    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b246367-cfbd-4a5e-9029-28b98dd6351f" (UID: "2b246367-cfbd-4a5e-9029-28b98dd6351f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:03:21.947745 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.947728    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-home" (OuterVolumeSpecName: "home") pod "2b246367-cfbd-4a5e-9029-28b98dd6351f" (UID: "2b246367-cfbd-4a5e-9029-28b98dd6351f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:03:21.947882 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.947798    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b246367-cfbd-4a5e-9029-28b98dd6351f" (UID: "2b246367-cfbd-4a5e-9029-28b98dd6351f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:03:21.947954 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.947909    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:03:21.947954 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.947938    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:03:21.947954 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.947947    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:03:21.949740 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.949714    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b246367-cfbd-4a5e-9029-28b98dd6351f-kube-api-access-x5j7k" (OuterVolumeSpecName: "kube-api-access-x5j7k") pod "2b246367-cfbd-4a5e-9029-28b98dd6351f" (UID: "2b246367-cfbd-4a5e-9029-28b98dd6351f"). InnerVolumeSpecName "kube-api-access-x5j7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:03:21.949852 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.949748    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b246367-cfbd-4a5e-9029-28b98dd6351f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b246367-cfbd-4a5e-9029-28b98dd6351f" (UID: "2b246367-cfbd-4a5e-9029-28b98dd6351f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:03:21.949852 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:21.949789    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-dshm" (OuterVolumeSpecName: "dshm") pod "2b246367-cfbd-4a5e-9029-28b98dd6351f" (UID: "2b246367-cfbd-4a5e-9029-28b98dd6351f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:03:22.048549 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:22.048473    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b246367-cfbd-4a5e-9029-28b98dd6351f-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:03:22.048549 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:22.048500    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5j7k\" (UniqueName: \"kubernetes.io/projected/2b246367-cfbd-4a5e-9029-28b98dd6351f-kube-api-access-x5j7k\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:03:22.048549 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:22.048511    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b246367-cfbd-4a5e-9029-28b98dd6351f-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:03:22.323985 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:22.323898    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"
Aug 01 00:03:22.323985 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:22.323907    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm" event={"ID":"2b246367-cfbd-4a5e-9029-28b98dd6351f","Type":"ContainerDied","Data":"ae7353156a1eb987f38a94ca23295d9f2f1a098e58e69ce26d89b0451d27b419"}
Aug 01 00:03:22.324446 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:22.324004    2551 scope.go:117] "RemoveContainer" containerID="02a54869f488d0c639c433a16efcd9713fe4724f9faf534842e553cb025b4894"
Aug 01 00:03:22.347177 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:22.347151    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"]
Aug 01 00:03:22.355187 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:22.355166    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6vkxm"]
Aug 01 00:03:23.947267 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:23.947233    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b246367-cfbd-4a5e-9029-28b98dd6351f" path="/var/lib/kubelet/pods/2b246367-cfbd-4a5e-9029-28b98dd6351f/volumes"
Aug 01 00:03:25.169326 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:25.169280    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2" podUID="a381c347-f60d-410b-9727-ed23a4faa15b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:03:25.410173 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:25.410133    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:25.472042 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:25.472008    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:35.169092 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:35.169049    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2" podUID="a381c347-f60d-410b-9727-ed23a4faa15b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:03:36.127059 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.127020    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2" podUID="a381c347-f60d-410b-9727-ed23a4faa15b" containerName="main" containerID="cri-o://5ef5bea21f5b89ba40690a4d21f9fe989a0777fd4621b125b0bbd5086dd51304" gracePeriod=45
Aug 01 00:03:36.369126 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.369101    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"
Aug 01 00:03:36.383826 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.383788    2551 generic.go:356] "Generic (PLEG): container finished" podID="a381c347-f60d-410b-9727-ed23a4faa15b" containerID="5ef5bea21f5b89ba40690a4d21f9fe989a0777fd4621b125b0bbd5086dd51304" exitCode=0
Aug 01 00:03:36.383988 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.383834    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"
Aug 01 00:03:36.383988 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.383828    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2" event={"ID":"a381c347-f60d-410b-9727-ed23a4faa15b","Type":"ContainerDied","Data":"5ef5bea21f5b89ba40690a4d21f9fe989a0777fd4621b125b0bbd5086dd51304"}
Aug 01 00:03:36.384107 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.383999    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2" event={"ID":"a381c347-f60d-410b-9727-ed23a4faa15b","Type":"ContainerDied","Data":"76c481c86936a26fe10901c00696fe3af32ddcdc1d9c1c117bb5feaae2db8089"}
Aug 01 00:03:36.384107 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.384046    2551 scope.go:117] "RemoveContainer" containerID="5ef5bea21f5b89ba40690a4d21f9fe989a0777fd4621b125b0bbd5086dd51304"
Aug 01 00:03:36.394962 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.394939    2551 scope.go:117] "RemoveContainer" containerID="5ef5bea21f5b89ba40690a4d21f9fe989a0777fd4621b125b0bbd5086dd51304"
Aug 01 00:03:36.395237 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:03:36.395211    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef5bea21f5b89ba40690a4d21f9fe989a0777fd4621b125b0bbd5086dd51304\": container with ID starting with 5ef5bea21f5b89ba40690a4d21f9fe989a0777fd4621b125b0bbd5086dd51304 not found: ID does not exist" containerID="5ef5bea21f5b89ba40690a4d21f9fe989a0777fd4621b125b0bbd5086dd51304"
Aug 01 00:03:36.395301 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.395249    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef5bea21f5b89ba40690a4d21f9fe989a0777fd4621b125b0bbd5086dd51304"} err="failed to get container status \"5ef5bea21f5b89ba40690a4d21f9fe989a0777fd4621b125b0bbd5086dd51304\": rpc error: code = NotFound desc = could not find container \"5ef5bea21f5b89ba40690a4d21f9fe989a0777fd4621b125b0bbd5086dd51304\": container with ID starting with 5ef5bea21f5b89ba40690a4d21f9fe989a0777fd4621b125b0bbd5086dd51304 not found: ID does not exist"
Aug 01 00:03:36.473317 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.473294    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a381c347-f60d-410b-9727-ed23a4faa15b-tls-certs\") pod \"a381c347-f60d-410b-9727-ed23a4faa15b\" (UID: \"a381c347-f60d-410b-9727-ed23a4faa15b\") "
Aug 01 00:03:36.473433 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.473383    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf8sw\" (UniqueName: \"kubernetes.io/projected/a381c347-f60d-410b-9727-ed23a4faa15b-kube-api-access-xf8sw\") pod \"a381c347-f60d-410b-9727-ed23a4faa15b\" (UID: \"a381c347-f60d-410b-9727-ed23a4faa15b\") "
Aug 01 00:03:36.475342 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.475319    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a381c347-f60d-410b-9727-ed23a4faa15b-kube-api-access-xf8sw" (OuterVolumeSpecName: "kube-api-access-xf8sw") pod "a381c347-f60d-410b-9727-ed23a4faa15b" (UID: "a381c347-f60d-410b-9727-ed23a4faa15b"). InnerVolumeSpecName "kube-api-access-xf8sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:03:36.475431 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.475345    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a381c347-f60d-410b-9727-ed23a4faa15b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a381c347-f60d-410b-9727-ed23a4faa15b" (UID: "a381c347-f60d-410b-9727-ed23a4faa15b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:03:36.574602 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.574524    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf8sw\" (UniqueName: \"kubernetes.io/projected/a381c347-f60d-410b-9727-ed23a4faa15b-kube-api-access-xf8sw\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:03:36.574602 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.574553    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a381c347-f60d-410b-9727-ed23a4faa15b-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:03:36.705945 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.705898    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"]
Aug 01 00:03:36.707883 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:36.707857    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587zpvp2"]
Aug 01 00:03:37.946396 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:37.946355    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a381c347-f60d-410b-9727-ed23a4faa15b" path="/var/lib/kubelet/pods/a381c347-f60d-410b-9727-ed23a4faa15b/volumes"
Aug 01 00:03:40.703701 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:40.703667    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:40.784187 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:40.784152    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:43.759366 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.759332    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"]
Aug 01 00:03:43.759754 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.759638    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b246367-cfbd-4a5e-9029-28b98dd6351f" containerName="main"
Aug 01 00:03:43.759754 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.759649    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b246367-cfbd-4a5e-9029-28b98dd6351f" containerName="main"
Aug 01 00:03:43.759754 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.759674    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a381c347-f60d-410b-9727-ed23a4faa15b" containerName="main"
Aug 01 00:03:43.759754 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.759679    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a381c347-f60d-410b-9727-ed23a4faa15b" containerName="main"
Aug 01 00:03:43.759754 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.759724    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a381c347-f60d-410b-9727-ed23a4faa15b" containerName="main"
Aug 01 00:03:43.759754 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.759732    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b246367-cfbd-4a5e-9029-28b98dd6351f" containerName="main"
Aug 01 00:03:43.771294 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.771268    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.772702 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.772678    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"]
Aug 01 00:03:43.773562 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.773518    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Aug 01 00:03:43.774195 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.774177    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Aug 01 00:03:43.774280 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.774247    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Aug 01 00:03:43.774397 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.774311    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-nm7mc\""
Aug 01 00:03:43.793024 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.792994    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:43.838406 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.838378    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.838516 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.838410    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.838516 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.838427    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.838591 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.838517    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l645q\" (UniqueName: \"kubernetes.io/projected/e72da00a-bce2-4d87-b5cd-10b9838d7107-kube-api-access-l645q\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.838591 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.838549    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.838591 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.838565    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72da00a-bce2-4d87-b5cd-10b9838d7107-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.939545 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.939513    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.939711 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.939556    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72da00a-bce2-4d87-b5cd-10b9838d7107-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.939711 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.939609    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.939711 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.939642    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.939879 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.939707    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.939879 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.939820    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l645q\" (UniqueName: \"kubernetes.io/projected/e72da00a-bce2-4d87-b5cd-10b9838d7107-kube-api-access-l645q\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.940004 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.939979    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.940058 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.939998    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.940203 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.940183    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.942090 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.942064    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.942365 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.942342    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72da00a-bce2-4d87-b5cd-10b9838d7107-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.950904 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.950883    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l645q\" (UniqueName: \"kubernetes.io/projected/e72da00a-bce2-4d87-b5cd-10b9838d7107-kube-api-access-l645q\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:43.964535 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.964511    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"]
Aug 01 00:03:43.968136 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.968118    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"
Aug 01 00:03:43.970008 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.969988    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-nd89l\""
Aug 01 00:03:43.977633 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.977615    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"]
Aug 01 00:03:43.994030 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:43.994007    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:44.040392 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:44.040314    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd6e6e6b-3225-4674-92c1-32dce2629fca-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh\" (UID: \"fd6e6e6b-3225-4674-92c1-32dce2629fca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"
Aug 01 00:03:44.040392 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:44.040357    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76ws8\" (UniqueName: \"kubernetes.io/projected/fd6e6e6b-3225-4674-92c1-32dce2629fca-kube-api-access-76ws8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh\" (UID: \"fd6e6e6b-3225-4674-92c1-32dce2629fca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"
Aug 01 00:03:44.082171 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:44.082136    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:44.141578 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:44.141534    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd6e6e6b-3225-4674-92c1-32dce2629fca-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh\" (UID: \"fd6e6e6b-3225-4674-92c1-32dce2629fca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"
Aug 01 00:03:44.142075 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:44.142050    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76ws8\" (UniqueName: \"kubernetes.io/projected/fd6e6e6b-3225-4674-92c1-32dce2629fca-kube-api-access-76ws8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh\" (UID: \"fd6e6e6b-3225-4674-92c1-32dce2629fca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"
Aug 01 00:03:44.145229 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:44.145198    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd6e6e6b-3225-4674-92c1-32dce2629fca-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh\" (UID: \"fd6e6e6b-3225-4674-92c1-32dce2629fca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"
Aug 01 00:03:44.150227 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:44.150185    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76ws8\" (UniqueName: \"kubernetes.io/projected/fd6e6e6b-3225-4674-92c1-32dce2629fca-kube-api-access-76ws8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh\" (UID: \"fd6e6e6b-3225-4674-92c1-32dce2629fca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"
Aug 01 00:03:44.213726 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:44.213693    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"]
Aug 01 00:03:44.217500 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:03:44.217476    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72da00a_bce2_4d87_b5cd_10b9838d7107.slice/crio-11622675ce3186d730e8fb6641c0701182050a7207307e5af06243ddb0f6cb89 WatchSource:0}: Error finding container 11622675ce3186d730e8fb6641c0701182050a7207307e5af06243ddb0f6cb89: Status 404 returned error can't find the container with id 11622675ce3186d730e8fb6641c0701182050a7207307e5af06243ddb0f6cb89
Aug 01 00:03:44.280730 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:44.280708    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"
Aug 01 00:03:44.419999 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:44.419967    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"]
Aug 01 00:03:44.423294 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:03:44.423257    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6e6e6b_3225_4674_92c1_32dce2629fca.slice/crio-9300070cc306ade3efaba6d10311fd39e0d457150cdb0976b5eb57fbb34dc495 WatchSource:0}: Error finding container 9300070cc306ade3efaba6d10311fd39e0d457150cdb0976b5eb57fbb34dc495: Status 404 returned error can't find the container with id 9300070cc306ade3efaba6d10311fd39e0d457150cdb0976b5eb57fbb34dc495
Aug 01 00:03:44.426980 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:44.426951    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf" event={"ID":"e72da00a-bce2-4d87-b5cd-10b9838d7107","Type":"ContainerStarted","Data":"1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361"}
Aug 01 00:03:44.427070 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:44.427007    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf" event={"ID":"e72da00a-bce2-4d87-b5cd-10b9838d7107","Type":"ContainerStarted","Data":"11622675ce3186d730e8fb6641c0701182050a7207307e5af06243ddb0f6cb89"}
Aug 01 00:03:44.447661 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:44.447614    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf" podStartSLOduration=1.447599957 podStartE2EDuration="1.447599957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:03:44.446276825 +0000 UTC m=+1067.140324241" watchObservedRunningTime="2026-08-01 00:03:44.447599957 +0000 UTC m=+1067.141647373"
Aug 01 00:03:45.082324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:45.082285    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:45.433895 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:45.433806    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh" event={"ID":"fd6e6e6b-3225-4674-92c1-32dce2629fca","Type":"ContainerStarted","Data":"fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f"}
Aug 01 00:03:45.433895 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:45.433871    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh" event={"ID":"fd6e6e6b-3225-4674-92c1-32dce2629fca","Type":"ContainerStarted","Data":"9300070cc306ade3efaba6d10311fd39e0d457150cdb0976b5eb57fbb34dc495"}
Aug 01 00:03:45.434202 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:45.433900    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"
Aug 01 00:03:45.438896 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:45.438870    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:45.452247 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:45.452199    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh" podStartSLOduration=2.4521838750000002 podStartE2EDuration="2.452183875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:03:45.450561894 +0000 UTC m=+1068.144609309" watchObservedRunningTime="2026-08-01 00:03:45.452183875 +0000 UTC m=+1068.146231290"
Aug 01 00:03:45.540208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:45.540173    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:47.390524 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:47.390493    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:50.379006 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:50.378871    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:51.385947 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:51.385897    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:53.022533 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:53.022501    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:53.087148 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:53.087110    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:54.083155 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:54.083107    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:54.095677 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:54.095648    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:54.482820 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:54.482793    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:03:57.563196 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:57.563164    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:57.662648 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:03:57.662607    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:04.770242 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:04.770198    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:16.439808 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:16.439775    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"
Aug 01 00:04:22.363814 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.363776    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:22.535213 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.535182    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"]
Aug 01 00:04:22.539682 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.539661    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"
Aug 01 00:04:22.544066 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.544044    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Aug 01 00:04:22.544066 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.544055    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-kd96r\""
Aug 01 00:04:22.544251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.544044    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Aug 01 00:04:22.544251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.544063    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-phjk2\""
Aug 01 00:04:22.544251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.544044    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Aug 01 00:04:22.553451 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.553429    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"]
Aug 01 00:04:22.593470 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.593444    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:22.681317 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.681217    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c580659a-f373-43d3-be3b-e28e4736ee78-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76\" (UID: \"c580659a-f373-43d3-be3b-e28e4736ee78\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"
Aug 01 00:04:22.681494 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.681338    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh9tg\" (UniqueName: \"kubernetes.io/projected/c580659a-f373-43d3-be3b-e28e4736ee78-kube-api-access-rh9tg\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76\" (UID: \"c580659a-f373-43d3-be3b-e28e4736ee78\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"
Aug 01 00:04:22.782449 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.782411    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh9tg\" (UniqueName: \"kubernetes.io/projected/c580659a-f373-43d3-be3b-e28e4736ee78-kube-api-access-rh9tg\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76\" (UID: \"c580659a-f373-43d3-be3b-e28e4736ee78\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"
Aug 01 00:04:22.782630 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.782485    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c580659a-f373-43d3-be3b-e28e4736ee78-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76\" (UID: \"c580659a-f373-43d3-be3b-e28e4736ee78\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"
Aug 01 00:04:22.785010 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.784974    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c580659a-f373-43d3-be3b-e28e4736ee78-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76\" (UID: \"c580659a-f373-43d3-be3b-e28e4736ee78\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"
Aug 01 00:04:22.790219 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.790193    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh9tg\" (UniqueName: \"kubernetes.io/projected/c580659a-f373-43d3-be3b-e28e4736ee78-kube-api-access-rh9tg\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76\" (UID: \"c580659a-f373-43d3-be3b-e28e4736ee78\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"
Aug 01 00:04:22.850751 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.850714    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"
Aug 01 00:04:22.984012 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:22.983968    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"]
Aug 01 00:04:22.986824 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:04:22.986798    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc580659a_f373_43d3_be3b_e28e4736ee78.slice/crio-600273aa4f7cdce625353eb2c924d85fa45d310b0d160f736e0407e7fb622718 WatchSource:0}: Error finding container 600273aa4f7cdce625353eb2c924d85fa45d310b0d160f736e0407e7fb622718: Status 404 returned error can't find the container with id 600273aa4f7cdce625353eb2c924d85fa45d310b0d160f736e0407e7fb622718
Aug 01 00:04:23.592783 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:23.592746    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76" event={"ID":"c580659a-f373-43d3-be3b-e28e4736ee78","Type":"ContainerStarted","Data":"e11eeca390a199fb8032608c5f72a97e8f67c3cd3f1a88cb4b31d3bb1a0954ec"}
Aug 01 00:04:23.593287 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:23.592806    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76" event={"ID":"c580659a-f373-43d3-be3b-e28e4736ee78","Type":"ContainerStarted","Data":"600273aa4f7cdce625353eb2c924d85fa45d310b0d160f736e0407e7fb622718"}
Aug 01 00:04:23.593287 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:23.592877    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"
Aug 01 00:04:23.610658 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:23.610614    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76" podStartSLOduration=1.610596752 podStartE2EDuration="1.610596752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:04:23.61002399 +0000 UTC m=+1106.304071407" watchObservedRunningTime="2026-08-01 00:04:23.610596752 +0000 UTC m=+1106.304644160"
Aug 01 00:04:28.372635 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:28.372598    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:29.948287 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:29.948257    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"]
Aug 01 00:04:29.958224 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:04:29.958184    2551 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:04:29.958224 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:04:29.958224 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:04:29.958224 ip-10-0-131-136 kubenswrapper[2551]:  > containerID="1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361" cmd=["/bin/sleep","15"]
Aug 01 00:04:29.958568 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:29.958241    2551 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf" message="" err=<
Aug 01 00:04:29.958568 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:04:29.958568 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:04:29.958568 ip-10-0-131-136 kubenswrapper[2551]:  >
Aug 01 00:04:29.958568 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:04:29.958258    2551 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:04:29.958568 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:04:29.958568 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:04:29.958568 ip-10-0-131-136 kubenswrapper[2551]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf" podUID="e72da00a-bce2-4d87-b5cd-10b9838d7107" containerName="main" containerID="cri-o://1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361"
Aug 01 00:04:29.958568 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:29.958312    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf" podUID="e72da00a-bce2-4d87-b5cd-10b9838d7107" containerName="main" containerID="cri-o://1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361" gracePeriod=60
Aug 01 00:04:29.958568 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:29.958399    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"]
Aug 01 00:04:29.969796 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:29.969770    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:30.063512 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.063483    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:30.336870 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.336844    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:04:30.446698 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.446669    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-model-cache\") pod \"e72da00a-bce2-4d87-b5cd-10b9838d7107\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") "
Aug 01 00:04:30.446854 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.446719    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-home\") pod \"e72da00a-bce2-4d87-b5cd-10b9838d7107\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") "
Aug 01 00:04:30.446854 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.446752    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-dshm\") pod \"e72da00a-bce2-4d87-b5cd-10b9838d7107\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") "
Aug 01 00:04:30.446854 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.446768    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72da00a-bce2-4d87-b5cd-10b9838d7107-tls-certs\") pod \"e72da00a-bce2-4d87-b5cd-10b9838d7107\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") "
Aug 01 00:04:30.446854 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.446802    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l645q\" (UniqueName: \"kubernetes.io/projected/e72da00a-bce2-4d87-b5cd-10b9838d7107-kube-api-access-l645q\") pod \"e72da00a-bce2-4d87-b5cd-10b9838d7107\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") "
Aug 01 00:04:30.446854 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.446828    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-tmp-dir\") pod \"e72da00a-bce2-4d87-b5cd-10b9838d7107\" (UID: \"e72da00a-bce2-4d87-b5cd-10b9838d7107\") "
Aug 01 00:04:30.447202 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.446956    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-model-cache" (OuterVolumeSpecName: "model-cache") pod "e72da00a-bce2-4d87-b5cd-10b9838d7107" (UID: "e72da00a-bce2-4d87-b5cd-10b9838d7107"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:04:30.447202 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.447040    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-home" (OuterVolumeSpecName: "home") pod "e72da00a-bce2-4d87-b5cd-10b9838d7107" (UID: "e72da00a-bce2-4d87-b5cd-10b9838d7107"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:04:30.447202 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.447094    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:04:30.447202 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.447161    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e72da00a-bce2-4d87-b5cd-10b9838d7107" (UID: "e72da00a-bce2-4d87-b5cd-10b9838d7107"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:04:30.448976 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.448958    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-dshm" (OuterVolumeSpecName: "dshm") pod "e72da00a-bce2-4d87-b5cd-10b9838d7107" (UID: "e72da00a-bce2-4d87-b5cd-10b9838d7107"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:04:30.449057 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.449014    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72da00a-bce2-4d87-b5cd-10b9838d7107-kube-api-access-l645q" (OuterVolumeSpecName: "kube-api-access-l645q") pod "e72da00a-bce2-4d87-b5cd-10b9838d7107" (UID: "e72da00a-bce2-4d87-b5cd-10b9838d7107"). InnerVolumeSpecName "kube-api-access-l645q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:04:30.449094 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.449082    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e72da00a-bce2-4d87-b5cd-10b9838d7107-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e72da00a-bce2-4d87-b5cd-10b9838d7107" (UID: "e72da00a-bce2-4d87-b5cd-10b9838d7107"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:04:30.548301 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.548234    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:04:30.548301 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.548259    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:04:30.548301 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.548270    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72da00a-bce2-4d87-b5cd-10b9838d7107-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:04:30.548301 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.548280    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l645q\" (UniqueName: \"kubernetes.io/projected/e72da00a-bce2-4d87-b5cd-10b9838d7107-kube-api-access-l645q\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:04:30.548301 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.548288    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72da00a-bce2-4d87-b5cd-10b9838d7107-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:04:30.627289 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.627252    2551 generic.go:356] "Generic (PLEG): container finished" podID="e72da00a-bce2-4d87-b5cd-10b9838d7107" containerID="1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361" exitCode=0
Aug 01 00:04:30.627450 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.627303    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"
Aug 01 00:04:30.627450 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.627341    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf" event={"ID":"e72da00a-bce2-4d87-b5cd-10b9838d7107","Type":"ContainerDied","Data":"1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361"}
Aug 01 00:04:30.627450 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.627408    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf" event={"ID":"e72da00a-bce2-4d87-b5cd-10b9838d7107","Type":"ContainerDied","Data":"11622675ce3186d730e8fb6641c0701182050a7207307e5af06243ddb0f6cb89"}
Aug 01 00:04:30.627450 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.627438    2551 scope.go:117] "RemoveContainer" containerID="1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361"
Aug 01 00:04:30.636259 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.636244    2551 scope.go:117] "RemoveContainer" containerID="1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361"
Aug 01 00:04:30.636517 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:04:30.636500    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361\": container with ID starting with 1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361 not found: ID does not exist" containerID="1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361"
Aug 01 00:04:30.636580 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.636523    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361"} err="failed to get container status \"1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361\": rpc error: code = NotFound desc = could not find container \"1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361\": container with ID starting with 1f61cff6b6243e165ebdf890cce321b9baab5bad78ecaf38c9e5a821aa466361 not found: ID does not exist"
Aug 01 00:04:30.658667 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.658636    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"]
Aug 01 00:04:30.669085 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:30.669064    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-8dsvf"]
Aug 01 00:04:31.947217 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:31.947181    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e72da00a-bce2-4d87-b5cd-10b9838d7107" path="/var/lib/kubelet/pods/e72da00a-bce2-4d87-b5cd-10b9838d7107/volumes"
Aug 01 00:04:36.440513 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:36.440431    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh" podUID="fd6e6e6b-3225-4674-92c1-32dce2629fca" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:04:40.755444 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:40.755400    2551 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh" message="" err="command '/bin/sleep 15' exited with 137: "
Aug 01 00:04:40.755782 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:04:40.755444    2551 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh" podUID="fd6e6e6b-3225-4674-92c1-32dce2629fca" containerName="main" containerID="cri-o://fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f"
Aug 01 00:04:40.755782 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:40.755489    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh" podUID="fd6e6e6b-3225-4674-92c1-32dce2629fca" containerName="main" containerID="cri-o://fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f" gracePeriod=50
Aug 01 00:04:40.783532 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:04:40.783499    2551 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6e6e6b_3225_4674_92c1_32dce2629fca.slice/crio-fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f.scope\": RecentStats: unable to find data in memory cache]"
Aug 01 00:04:40.889315 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:40.889292    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh_fd6e6e6b-3225-4674-92c1-32dce2629fca/main/0.log"
Aug 01 00:04:40.891549 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:40.891533    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"
Aug 01 00:04:40.927361 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:40.927331    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd6e6e6b-3225-4674-92c1-32dce2629fca-tls-certs\") pod \"fd6e6e6b-3225-4674-92c1-32dce2629fca\" (UID: \"fd6e6e6b-3225-4674-92c1-32dce2629fca\") "
Aug 01 00:04:40.927494 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:40.927413    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76ws8\" (UniqueName: \"kubernetes.io/projected/fd6e6e6b-3225-4674-92c1-32dce2629fca-kube-api-access-76ws8\") pod \"fd6e6e6b-3225-4674-92c1-32dce2629fca\" (UID: \"fd6e6e6b-3225-4674-92c1-32dce2629fca\") "
Aug 01 00:04:40.929384 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:40.929352    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6e6e6b-3225-4674-92c1-32dce2629fca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd6e6e6b-3225-4674-92c1-32dce2629fca" (UID: "fd6e6e6b-3225-4674-92c1-32dce2629fca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:04:40.929504 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:40.929473    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6e6e6b-3225-4674-92c1-32dce2629fca-kube-api-access-76ws8" (OuterVolumeSpecName: "kube-api-access-76ws8") pod "fd6e6e6b-3225-4674-92c1-32dce2629fca" (UID: "fd6e6e6b-3225-4674-92c1-32dce2629fca"). InnerVolumeSpecName "kube-api-access-76ws8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:04:41.028547 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:41.028472    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76ws8\" (UniqueName: \"kubernetes.io/projected/fd6e6e6b-3225-4674-92c1-32dce2629fca-kube-api-access-76ws8\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:04:41.028547 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:41.028496    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd6e6e6b-3225-4674-92c1-32dce2629fca-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:04:41.676223 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:41.676196    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh_fd6e6e6b-3225-4674-92c1-32dce2629fca/main/0.log"
Aug 01 00:04:41.676407 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:41.676239    2551 generic.go:356] "Generic (PLEG): container finished" podID="fd6e6e6b-3225-4674-92c1-32dce2629fca" containerID="fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f" exitCode=1
Aug 01 00:04:41.676407 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:41.676280    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh" event={"ID":"fd6e6e6b-3225-4674-92c1-32dce2629fca","Type":"ContainerDied","Data":"fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f"}
Aug 01 00:04:41.676407 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:41.676296    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"
Aug 01 00:04:41.676407 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:41.676318    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh" event={"ID":"fd6e6e6b-3225-4674-92c1-32dce2629fca","Type":"ContainerDied","Data":"9300070cc306ade3efaba6d10311fd39e0d457150cdb0976b5eb57fbb34dc495"}
Aug 01 00:04:41.676407 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:41.676339    2551 scope.go:117] "RemoveContainer" containerID="fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f"
Aug 01 00:04:41.684812 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:41.684794    2551 scope.go:117] "RemoveContainer" containerID="fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f"
Aug 01 00:04:41.685075 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:04:41.685051    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f\": container with ID starting with fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f not found: ID does not exist" containerID="fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f"
Aug 01 00:04:41.685183 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:41.685081    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f"} err="failed to get container status \"fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f\": rpc error: code = NotFound desc = could not find container \"fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f\": container with ID starting with fd909c3292140fc891262891f5d9a0c120acea08f0e55aefc83fca9f343d1c1f not found: ID does not exist"
Aug 01 00:04:41.695707 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:41.695684    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"]
Aug 01 00:04:41.700152 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:41.700133    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfvctwh"]
Aug 01 00:04:41.946808 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:41.946730    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd6e6e6b-3225-4674-92c1-32dce2629fca" path="/var/lib/kubelet/pods/fd6e6e6b-3225-4674-92c1-32dce2629fca/volumes"
Aug 01 00:04:51.016537 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:51.016444    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:51.112829 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:51.112783    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:54.598270 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:54.598241    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"
Aug 01 00:04:56.926627 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:04:56.926589    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:01.097700 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.097664    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:01.285969 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.285937    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"]
Aug 01 00:05:01.286251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.286239    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e72da00a-bce2-4d87-b5cd-10b9838d7107" containerName="main"
Aug 01 00:05:01.286251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.286252    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e72da00a-bce2-4d87-b5cd-10b9838d7107" containerName="main"
Aug 01 00:05:01.286348 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.286265    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd6e6e6b-3225-4674-92c1-32dce2629fca" containerName="main"
Aug 01 00:05:01.286348 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.286271    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd6e6e6b-3225-4674-92c1-32dce2629fca" containerName="main"
Aug 01 00:05:01.286348 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.286322    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd6e6e6b-3225-4674-92c1-32dce2629fca" containerName="main"
Aug 01 00:05:01.286348 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.286334    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e72da00a-bce2-4d87-b5cd-10b9838d7107" containerName="main"
Aug 01 00:05:01.290548 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.290530    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.293556 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.293531    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Aug 01 00:05:01.293662 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.293531    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-n8jf8\""
Aug 01 00:05:01.293662 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.293641    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Aug 01 00:05:01.293770 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.293755    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-lcdzv\""
Aug 01 00:05:01.293861 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.293847    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Aug 01 00:05:01.298065 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.298040    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"]
Aug 01 00:05:01.310296 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.310262    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:01.399442 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.399349    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3619e7-a09d-4fdd-aef7-8231248544e3-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kj7fp\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.399442 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.399409    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3619e7-a09d-4fdd-aef7-8231248544e3-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kj7fp\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.399674 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.399489    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3619e7-a09d-4fdd-aef7-8231248544e3-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kj7fp\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.399674 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.399534    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gp8r\" (UniqueName: \"kubernetes.io/projected/9c3619e7-a09d-4fdd-aef7-8231248544e3-kube-api-access-5gp8r\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kj7fp\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.500626 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.500571    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3619e7-a09d-4fdd-aef7-8231248544e3-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kj7fp\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.500817 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.500662    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3619e7-a09d-4fdd-aef7-8231248544e3-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kj7fp\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.500817 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.500700    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gp8r\" (UniqueName: \"kubernetes.io/projected/9c3619e7-a09d-4fdd-aef7-8231248544e3-kube-api-access-5gp8r\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kj7fp\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.500817 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.500776    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3619e7-a09d-4fdd-aef7-8231248544e3-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kj7fp\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.501141 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.501119    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3619e7-a09d-4fdd-aef7-8231248544e3-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kj7fp\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.501228 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.501161    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3619e7-a09d-4fdd-aef7-8231248544e3-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kj7fp\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.503434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.503407    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3619e7-a09d-4fdd-aef7-8231248544e3-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kj7fp\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.509256 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.509229    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gp8r\" (UniqueName: \"kubernetes.io/projected/9c3619e7-a09d-4fdd-aef7-8231248544e3-kube-api-access-5gp8r\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795kj7fp\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.601443 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.601404    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:01.735407 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.735381    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"]
Aug 01 00:05:01.737442 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:05:01.737413    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3619e7_a09d_4fdd_aef7_8231248544e3.slice/crio-7d9b24ba79bf38015f092d7b930e7c66afee6921e4f011a60412f8c409e4bd70 WatchSource:0}: Error finding container 7d9b24ba79bf38015f092d7b930e7c66afee6921e4f011a60412f8c409e4bd70: Status 404 returned error can't find the container with id 7d9b24ba79bf38015f092d7b930e7c66afee6921e4f011a60412f8c409e4bd70
Aug 01 00:05:01.765368 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:01.765336    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp" event={"ID":"9c3619e7-a09d-4fdd-aef7-8231248544e3","Type":"ContainerStarted","Data":"7d9b24ba79bf38015f092d7b930e7c66afee6921e4f011a60412f8c409e4bd70"}
Aug 01 00:05:02.771231 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:02.771196    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp" event={"ID":"9c3619e7-a09d-4fdd-aef7-8231248544e3","Type":"ContainerStarted","Data":"4ed3e08d45c2bf6af09d301c8758ae472f1bd233db961889cb23c4134aff0d7a"}
Aug 01 00:05:02.771625 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:02.771308    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:02.787672 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:02.787614    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp" podStartSLOduration=1.787598579 podStartE2EDuration="1.787598579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:05:02.786005417 +0000 UTC m=+1145.480052850" watchObservedRunningTime="2026-08-01 00:05:02.787598579 +0000 UTC m=+1145.481645995"
Aug 01 00:05:02.961151 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:02.961114    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:03.060890 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:03.060798    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:10.507471 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:10.507435    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:33.777459 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:33.777427    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:34.787648 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:34.787604    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:34.788526 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:34.788506    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"]
Aug 01 00:05:34.882747 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:34.882705    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:43.778213 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:43.778177    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp" podUID="9c3619e7-a09d-4fdd-aef7-8231248544e3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:05:49.811199 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:49.811157    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp" podUID="9c3619e7-a09d-4fdd-aef7-8231248544e3" containerName="main" containerID="cri-o://4ed3e08d45c2bf6af09d301c8758ae472f1bd233db961889cb23c4134aff0d7a" gracePeriod=45
Aug 01 00:05:49.963317 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:49.963278    2551 generic.go:356] "Generic (PLEG): container finished" podID="9c3619e7-a09d-4fdd-aef7-8231248544e3" containerID="4ed3e08d45c2bf6af09d301c8758ae472f1bd233db961889cb23c4134aff0d7a" exitCode=0
Aug 01 00:05:49.963465 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:49.963351    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp" event={"ID":"9c3619e7-a09d-4fdd-aef7-8231248544e3","Type":"ContainerDied","Data":"4ed3e08d45c2bf6af09d301c8758ae472f1bd233db961889cb23c4134aff0d7a"}
Aug 01 00:05:50.056047 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.056021    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:50.115532 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.115446    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gp8r\" (UniqueName: \"kubernetes.io/projected/9c3619e7-a09d-4fdd-aef7-8231248544e3-kube-api-access-5gp8r\") pod \"9c3619e7-a09d-4fdd-aef7-8231248544e3\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") "
Aug 01 00:05:50.115532 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.115483    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3619e7-a09d-4fdd-aef7-8231248544e3-tls-certs\") pod \"9c3619e7-a09d-4fdd-aef7-8231248544e3\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") "
Aug 01 00:05:50.115714 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.115544    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3619e7-a09d-4fdd-aef7-8231248544e3-tokenizer-cache\") pod \"9c3619e7-a09d-4fdd-aef7-8231248544e3\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") "
Aug 01 00:05:50.115714 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.115602    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3619e7-a09d-4fdd-aef7-8231248544e3-cachi2-cache\") pod \"9c3619e7-a09d-4fdd-aef7-8231248544e3\" (UID: \"9c3619e7-a09d-4fdd-aef7-8231248544e3\") "
Aug 01 00:05:50.115901 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.115874    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3619e7-a09d-4fdd-aef7-8231248544e3-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "9c3619e7-a09d-4fdd-aef7-8231248544e3" (UID: "9c3619e7-a09d-4fdd-aef7-8231248544e3"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:05:50.115981 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.115904    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3619e7-a09d-4fdd-aef7-8231248544e3-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "9c3619e7-a09d-4fdd-aef7-8231248544e3" (UID: "9c3619e7-a09d-4fdd-aef7-8231248544e3"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:05:50.117735 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.117713    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3619e7-a09d-4fdd-aef7-8231248544e3-kube-api-access-5gp8r" (OuterVolumeSpecName: "kube-api-access-5gp8r") pod "9c3619e7-a09d-4fdd-aef7-8231248544e3" (UID: "9c3619e7-a09d-4fdd-aef7-8231248544e3"). InnerVolumeSpecName "kube-api-access-5gp8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:05:50.117735 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.117722    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c3619e7-a09d-4fdd-aef7-8231248544e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c3619e7-a09d-4fdd-aef7-8231248544e3" (UID: "9c3619e7-a09d-4fdd-aef7-8231248544e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:05:50.216902 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.216868    2551 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3619e7-a09d-4fdd-aef7-8231248544e3-cachi2-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:05:50.216902 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.216897    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gp8r\" (UniqueName: \"kubernetes.io/projected/9c3619e7-a09d-4fdd-aef7-8231248544e3-kube-api-access-5gp8r\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:05:50.216902 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.216907    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3619e7-a09d-4fdd-aef7-8231248544e3-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:05:50.217151 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.216937    2551 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3619e7-a09d-4fdd-aef7-8231248544e3-tokenizer-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:05:50.968863 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.968829    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"
Aug 01 00:05:50.968863 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.968849    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp" event={"ID":"9c3619e7-a09d-4fdd-aef7-8231248544e3","Type":"ContainerDied","Data":"7d9b24ba79bf38015f092d7b930e7c66afee6921e4f011a60412f8c409e4bd70"}
Aug 01 00:05:50.969450 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.968902    2551 scope.go:117] "RemoveContainer" containerID="4ed3e08d45c2bf6af09d301c8758ae472f1bd233db961889cb23c4134aff0d7a"
Aug 01 00:05:50.987795 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.987773    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"]
Aug 01 00:05:50.991894 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:50.991871    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795kj7fp"]
Aug 01 00:05:51.946134 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:51.946103    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c3619e7-a09d-4fdd-aef7-8231248544e3" path="/var/lib/kubelet/pods/9c3619e7-a09d-4fdd-aef7-8231248544e3/volumes"
Aug 01 00:05:54.800838 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:54.800785    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:54.899164 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:05:54.899136    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:00.614016 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:00.613981    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:06.074402 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:06.074363    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:06.133713 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:06.133675    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:06.903407 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:06.903354    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"]
Aug 01 00:06:06.903858 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:06.903840    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c3619e7-a09d-4fdd-aef7-8231248544e3" containerName="main"
Aug 01 00:06:06.903858 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:06.903858    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c3619e7-a09d-4fdd-aef7-8231248544e3" containerName="main"
Aug 01 00:06:06.904053 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:06.903976    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c3619e7-a09d-4fdd-aef7-8231248544e3" containerName="main"
Aug 01 00:06:06.908280 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:06.908259    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:06.910373 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:06.910352    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-lvsfb\""
Aug 01 00:06:06.910506 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:06.910377    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Aug 01 00:06:06.911062 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:06.911040    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Aug 01 00:06:06.911166 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:06.911106    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Aug 01 00:06:06.917030 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:06.917003    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"]
Aug 01 00:06:06.950660 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:06.950630    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:07.068461 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.068430    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrs9\" (UniqueName: \"kubernetes.io/projected/42d4d01b-82c0-44c4-81c5-89dd713ec82e-kube-api-access-lcrs9\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.068613 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.068466    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.068613 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.068506    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.068613 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.068532    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d4d01b-82c0-44c4-81c5-89dd713ec82e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.068613 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.068604    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.068751 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.068661    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.140094 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.140060    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"]
Aug 01 00:06:07.143260 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.143242    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"
Aug 01 00:06:07.145627 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.145603    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-2r9vb\""
Aug 01 00:06:07.153034 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.153008    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"]
Aug 01 00:06:07.169246 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.169187    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d4d01b-82c0-44c4-81c5-89dd713ec82e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.169246 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.169242    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.169400 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.169279    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.169400 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.169299    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrs9\" (UniqueName: \"kubernetes.io/projected/42d4d01b-82c0-44c4-81c5-89dd713ec82e-kube-api-access-lcrs9\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.169400 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.169318    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.169538 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.169434    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.169638 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.169618    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.169737 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.169719    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.169794 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.169750    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.171673 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.171634    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.171776 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.171742    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d4d01b-82c0-44c4-81c5-89dd713ec82e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.172112 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.172088    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:07.177349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.177325    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrs9\" (UniqueName: \"kubernetes.io/projected/42d4d01b-82c0-44c4-81c5-89dd713ec82e-kube-api-access-lcrs9\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.220883 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.220852    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:07.270311 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.270017    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5bs7\" (UniqueName: \"kubernetes.io/projected/33ef33f4-edc2-4996-9a67-9e369e1d98e9-kube-api-access-t5bs7\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d\" (UID: \"33ef33f4-edc2-4996-9a67-9e369e1d98e9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"
Aug 01 00:06:07.270311 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.270087    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ef33f4-edc2-4996-9a67-9e369e1d98e9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d\" (UID: \"33ef33f4-edc2-4996-9a67-9e369e1d98e9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"
Aug 01 00:06:07.350309 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.350141    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"]
Aug 01 00:06:07.353005 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:06:07.352970    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d4d01b_82c0_44c4_81c5_89dd713ec82e.slice/crio-0dcc245f3ed2353a2a7a4deb74585492b94bdea89249d162c5a47816cd796185 WatchSource:0}: Error finding container 0dcc245f3ed2353a2a7a4deb74585492b94bdea89249d162c5a47816cd796185: Status 404 returned error can't find the container with id 0dcc245f3ed2353a2a7a4deb74585492b94bdea89249d162c5a47816cd796185
Aug 01 00:06:07.371114 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.371093    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ef33f4-edc2-4996-9a67-9e369e1d98e9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d\" (UID: \"33ef33f4-edc2-4996-9a67-9e369e1d98e9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"
Aug 01 00:06:07.371214 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.371185    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5bs7\" (UniqueName: \"kubernetes.io/projected/33ef33f4-edc2-4996-9a67-9e369e1d98e9-kube-api-access-t5bs7\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d\" (UID: \"33ef33f4-edc2-4996-9a67-9e369e1d98e9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"
Aug 01 00:06:07.373533 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.373507    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ef33f4-edc2-4996-9a67-9e369e1d98e9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d\" (UID: \"33ef33f4-edc2-4996-9a67-9e369e1d98e9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"
Aug 01 00:06:07.379469 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.379450    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5bs7\" (UniqueName: \"kubernetes.io/projected/33ef33f4-edc2-4996-9a67-9e369e1d98e9-kube-api-access-t5bs7\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d\" (UID: \"33ef33f4-edc2-4996-9a67-9e369e1d98e9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"
Aug 01 00:06:07.453571 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.453507    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"
Aug 01 00:06:07.576815 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:07.576790    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"]
Aug 01 00:06:07.579186 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:06:07.579155    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ef33f4_edc2_4996_9a67_9e369e1d98e9.slice/crio-e13af1ecddd1e7dae6aa20ec5cc64870e563348a732071506774a52e329f6df5 WatchSource:0}: Error finding container e13af1ecddd1e7dae6aa20ec5cc64870e563348a732071506774a52e329f6df5: Status 404 returned error can't find the container with id e13af1ecddd1e7dae6aa20ec5cc64870e563348a732071506774a52e329f6df5
Aug 01 00:06:08.036871 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:08.036832    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d" event={"ID":"33ef33f4-edc2-4996-9a67-9e369e1d98e9","Type":"ContainerStarted","Data":"fc30dcd3d06b38595780b3d95946744992a503352c55a2b960f0469bb045c56d"}
Aug 01 00:06:08.037085 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:08.036896    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d" event={"ID":"33ef33f4-edc2-4996-9a67-9e369e1d98e9","Type":"ContainerStarted","Data":"e13af1ecddd1e7dae6aa20ec5cc64870e563348a732071506774a52e329f6df5"}
Aug 01 00:06:08.037085 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:08.036940    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"
Aug 01 00:06:08.038885 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:08.038855    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9" event={"ID":"42d4d01b-82c0-44c4-81c5-89dd713ec82e","Type":"ContainerStarted","Data":"7458065178e79c33be403f03559d7de6777d9e545729e93e11d988c00c4fe9ff"}
Aug 01 00:06:08.039014 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:08.038909    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9" event={"ID":"42d4d01b-82c0-44c4-81c5-89dd713ec82e","Type":"ContainerStarted","Data":"0dcc245f3ed2353a2a7a4deb74585492b94bdea89249d162c5a47816cd796185"}
Aug 01 00:06:08.057946 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:08.057292    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d" podStartSLOduration=1.057268426 podStartE2EDuration="1.057268426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:06:08.054974232 +0000 UTC m=+1210.749021649" watchObservedRunningTime="2026-08-01 00:06:08.057268426 +0000 UTC m=+1210.751315843"
Aug 01 00:06:08.073204 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:08.073147    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9" podStartSLOduration=2.07312599 podStartE2EDuration="2.07312599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:06:08.070466819 +0000 UTC m=+1210.764514236" watchObservedRunningTime="2026-08-01 00:06:08.07312599 +0000 UTC m=+1210.767173408"
Aug 01 00:06:08.221975 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:08.221930    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:17.221913 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:17.221874    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:17.234283 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:17.234256    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:17.891739 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:17.891708    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:18.097862 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:18.097834    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:24.183944 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:24.183892    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"]
Aug 01 00:06:24.189159 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:24.189127    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:24.290785 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:24.290745    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:24.598450 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:24.598415    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76" podUID="c580659a-f373-43d3-be3b-e28e4736ee78" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:06:34.598579 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:34.598544    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76" podUID="c580659a-f373-43d3-be3b-e28e4736ee78" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:06:39.045005 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:39.044968    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"
Aug 01 00:06:39.203500 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:39.203457    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76" podUID="c580659a-f373-43d3-be3b-e28e4736ee78" containerName="main" containerID="cri-o://e11eeca390a199fb8032608c5f72a97e8f67c3cd3f1a88cb4b31d3bb1a0954ec" gracePeriod=45
Aug 01 00:06:39.451074 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:39.451043    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"
Aug 01 00:06:39.530327 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:39.530299    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c580659a-f373-43d3-be3b-e28e4736ee78-tls-certs\") pod \"c580659a-f373-43d3-be3b-e28e4736ee78\" (UID: \"c580659a-f373-43d3-be3b-e28e4736ee78\") "
Aug 01 00:06:39.530493 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:39.530357    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh9tg\" (UniqueName: \"kubernetes.io/projected/c580659a-f373-43d3-be3b-e28e4736ee78-kube-api-access-rh9tg\") pod \"c580659a-f373-43d3-be3b-e28e4736ee78\" (UID: \"c580659a-f373-43d3-be3b-e28e4736ee78\") "
Aug 01 00:06:39.532489 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:39.532463    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c580659a-f373-43d3-be3b-e28e4736ee78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c580659a-f373-43d3-be3b-e28e4736ee78" (UID: "c580659a-f373-43d3-be3b-e28e4736ee78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:06:39.532592 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:39.532502    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c580659a-f373-43d3-be3b-e28e4736ee78-kube-api-access-rh9tg" (OuterVolumeSpecName: "kube-api-access-rh9tg") pod "c580659a-f373-43d3-be3b-e28e4736ee78" (UID: "c580659a-f373-43d3-be3b-e28e4736ee78"). InnerVolumeSpecName "kube-api-access-rh9tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:06:39.631876 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:39.631788    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh9tg\" (UniqueName: \"kubernetes.io/projected/c580659a-f373-43d3-be3b-e28e4736ee78-kube-api-access-rh9tg\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:06:39.631876 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:39.631832    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c580659a-f373-43d3-be3b-e28e4736ee78-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:06:40.178696 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.178665    2551 generic.go:356] "Generic (PLEG): container finished" podID="c580659a-f373-43d3-be3b-e28e4736ee78" containerID="e11eeca390a199fb8032608c5f72a97e8f67c3cd3f1a88cb4b31d3bb1a0954ec" exitCode=0
Aug 01 00:06:40.179136 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.178701    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76" event={"ID":"c580659a-f373-43d3-be3b-e28e4736ee78","Type":"ContainerDied","Data":"e11eeca390a199fb8032608c5f72a97e8f67c3cd3f1a88cb4b31d3bb1a0954ec"}
Aug 01 00:06:40.179136 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.178727    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"
Aug 01 00:06:40.179136 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.178745    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76" event={"ID":"c580659a-f373-43d3-be3b-e28e4736ee78","Type":"ContainerDied","Data":"600273aa4f7cdce625353eb2c924d85fa45d310b0d160f736e0407e7fb622718"}
Aug 01 00:06:40.179136 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.178767    2551 scope.go:117] "RemoveContainer" containerID="e11eeca390a199fb8032608c5f72a97e8f67c3cd3f1a88cb4b31d3bb1a0954ec"
Aug 01 00:06:40.186995 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.186975    2551 scope.go:117] "RemoveContainer" containerID="e11eeca390a199fb8032608c5f72a97e8f67c3cd3f1a88cb4b31d3bb1a0954ec"
Aug 01 00:06:40.187242 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:06:40.187221    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11eeca390a199fb8032608c5f72a97e8f67c3cd3f1a88cb4b31d3bb1a0954ec\": container with ID starting with e11eeca390a199fb8032608c5f72a97e8f67c3cd3f1a88cb4b31d3bb1a0954ec not found: ID does not exist" containerID="e11eeca390a199fb8032608c5f72a97e8f67c3cd3f1a88cb4b31d3bb1a0954ec"
Aug 01 00:06:40.187296 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.187249    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11eeca390a199fb8032608c5f72a97e8f67c3cd3f1a88cb4b31d3bb1a0954ec"} err="failed to get container status \"e11eeca390a199fb8032608c5f72a97e8f67c3cd3f1a88cb4b31d3bb1a0954ec\": rpc error: code = NotFound desc = could not find container \"e11eeca390a199fb8032608c5f72a97e8f67c3cd3f1a88cb4b31d3bb1a0954ec\": container with ID starting with e11eeca390a199fb8032608c5f72a97e8f67c3cd3f1a88cb4b31d3bb1a0954ec not found: ID does not exist"
Aug 01 00:06:40.194628 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.194606    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"]
Aug 01 00:06:40.199385 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.199365    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcwxv76"]
Aug 01 00:06:40.605695 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.605665    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:40.623447 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.623417    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"]
Aug 01 00:06:40.624879 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.624855    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"]
Aug 01 00:06:40.633737 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:06:40.633702    2551 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:06:40.633737 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:06:40.633737 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:06:40.633737 ip-10-0-131-136 kubenswrapper[2551]:  > containerID="7458065178e79c33be403f03559d7de6777d9e545729e93e11d988c00c4fe9ff" cmd=["/bin/sleep","15"]
Aug 01 00:06:40.634045 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.633754    2551 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9" message="" err=<
Aug 01 00:06:40.634045 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:06:40.634045 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:06:40.634045 ip-10-0-131-136 kubenswrapper[2551]:  >
Aug 01 00:06:40.634045 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:06:40.633766    2551 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:06:40.634045 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:06:40.634045 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:06:40.634045 ip-10-0-131-136 kubenswrapper[2551]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9" podUID="42d4d01b-82c0-44c4-81c5-89dd713ec82e" containerName="main" containerID="cri-o://7458065178e79c33be403f03559d7de6777d9e545729e93e11d988c00c4fe9ff"
Aug 01 00:06:40.634045 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.633809    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9" podUID="42d4d01b-82c0-44c4-81c5-89dd713ec82e" containerName="main" containerID="cri-o://7458065178e79c33be403f03559d7de6777d9e545729e93e11d988c00c4fe9ff" gracePeriod=60
Aug 01 00:06:40.683406 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:40.683381    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:41.086298 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.086264    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9" podUID="42d4d01b-82c0-44c4-81c5-89dd713ec82e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Aug 01 00:06:41.186080 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.186049    2551 generic.go:356] "Generic (PLEG): container finished" podID="42d4d01b-82c0-44c4-81c5-89dd713ec82e" containerID="7458065178e79c33be403f03559d7de6777d9e545729e93e11d988c00c4fe9ff" exitCode=0
Aug 01 00:06:41.186441 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.186123    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9" event={"ID":"42d4d01b-82c0-44c4-81c5-89dd713ec82e","Type":"ContainerDied","Data":"7458065178e79c33be403f03559d7de6777d9e545729e93e11d988c00c4fe9ff"}
Aug 01 00:06:41.378893 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.378866    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:41.447273 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.447244    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcrs9\" (UniqueName: \"kubernetes.io/projected/42d4d01b-82c0-44c4-81c5-89dd713ec82e-kube-api-access-lcrs9\") pod \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") "
Aug 01 00:06:41.447426 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.447293    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-tmp-dir\") pod \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") "
Aug 01 00:06:41.447426 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.447317    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-dshm\") pod \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") "
Aug 01 00:06:41.447426 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.447370    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d4d01b-82c0-44c4-81c5-89dd713ec82e-tls-certs\") pod \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") "
Aug 01 00:06:41.447426 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.447392    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-home\") pod \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") "
Aug 01 00:06:41.447645 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.447428    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-model-cache\") pod \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\" (UID: \"42d4d01b-82c0-44c4-81c5-89dd713ec82e\") "
Aug 01 00:06:41.447645 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.447569    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42d4d01b-82c0-44c4-81c5-89dd713ec82e" (UID: "42d4d01b-82c0-44c4-81c5-89dd713ec82e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:41.447747 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.447677    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-home" (OuterVolumeSpecName: "home") pod "42d4d01b-82c0-44c4-81c5-89dd713ec82e" (UID: "42d4d01b-82c0-44c4-81c5-89dd713ec82e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:41.447747 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.447736    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:06:41.447815 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.447767    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-model-cache" (OuterVolumeSpecName: "model-cache") pod "42d4d01b-82c0-44c4-81c5-89dd713ec82e" (UID: "42d4d01b-82c0-44c4-81c5-89dd713ec82e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:41.449562 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.449529    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42d4d01b-82c0-44c4-81c5-89dd713ec82e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42d4d01b-82c0-44c4-81c5-89dd713ec82e" (UID: "42d4d01b-82c0-44c4-81c5-89dd713ec82e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:06:41.449562 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.449538    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-dshm" (OuterVolumeSpecName: "dshm") pod "42d4d01b-82c0-44c4-81c5-89dd713ec82e" (UID: "42d4d01b-82c0-44c4-81c5-89dd713ec82e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:06:41.449707 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.449593    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d4d01b-82c0-44c4-81c5-89dd713ec82e-kube-api-access-lcrs9" (OuterVolumeSpecName: "kube-api-access-lcrs9") pod "42d4d01b-82c0-44c4-81c5-89dd713ec82e" (UID: "42d4d01b-82c0-44c4-81c5-89dd713ec82e"). InnerVolumeSpecName "kube-api-access-lcrs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:06:41.548434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.548398    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:06:41.548434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.548428    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d4d01b-82c0-44c4-81c5-89dd713ec82e-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:06:41.548434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.548438    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:06:41.548658 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.548446    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d4d01b-82c0-44c4-81c5-89dd713ec82e-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:06:41.548658 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.548455    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcrs9\" (UniqueName: \"kubernetes.io/projected/42d4d01b-82c0-44c4-81c5-89dd713ec82e-kube-api-access-lcrs9\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:06:41.946798 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:41.946766    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c580659a-f373-43d3-be3b-e28e4736ee78" path="/var/lib/kubelet/pods/c580659a-f373-43d3-be3b-e28e4736ee78/volumes"
Aug 01 00:06:42.191841 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:42.191805    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9" event={"ID":"42d4d01b-82c0-44c4-81c5-89dd713ec82e","Type":"ContainerDied","Data":"0dcc245f3ed2353a2a7a4deb74585492b94bdea89249d162c5a47816cd796185"}
Aug 01 00:06:42.192342 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:42.191845    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"
Aug 01 00:06:42.192342 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:42.191872    2551 scope.go:117] "RemoveContainer" containerID="7458065178e79c33be403f03559d7de6777d9e545729e93e11d988c00c4fe9ff"
Aug 01 00:06:42.214947 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:42.214896    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"]
Aug 01 00:06:42.215101 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:42.214967    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-wmld9"]
Aug 01 00:06:43.946899 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:43.946864    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42d4d01b-82c0-44c4-81c5-89dd713ec82e" path="/var/lib/kubelet/pods/42d4d01b-82c0-44c4-81c5-89dd713ec82e/volumes"
Aug 01 00:06:44.266395 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:44.266348    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:44.380650 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:44.380600    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:49.045382 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:49.045343    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d" podUID="33ef33f4-edc2-4996-9a67-9e369e1d98e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:06:50.416310 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:50.416233    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:55.645696 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:55.645646    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d" podUID="33ef33f4-edc2-4996-9a67-9e369e1d98e9" containerName="main" containerID="cri-o://fc30dcd3d06b38595780b3d95946744992a503352c55a2b960f0469bb045c56d" gracePeriod=45
Aug 01 00:06:55.885521 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:55.885496    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"
Aug 01 00:06:55.975098 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:55.975071    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ef33f4-edc2-4996-9a67-9e369e1d98e9-tls-certs\") pod \"33ef33f4-edc2-4996-9a67-9e369e1d98e9\" (UID: \"33ef33f4-edc2-4996-9a67-9e369e1d98e9\") "
Aug 01 00:06:55.975274 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:55.975255    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5bs7\" (UniqueName: \"kubernetes.io/projected/33ef33f4-edc2-4996-9a67-9e369e1d98e9-kube-api-access-t5bs7\") pod \"33ef33f4-edc2-4996-9a67-9e369e1d98e9\" (UID: \"33ef33f4-edc2-4996-9a67-9e369e1d98e9\") "
Aug 01 00:06:55.977219 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:55.977192    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ef33f4-edc2-4996-9a67-9e369e1d98e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33ef33f4-edc2-4996-9a67-9e369e1d98e9" (UID: "33ef33f4-edc2-4996-9a67-9e369e1d98e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:06:55.977330 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:55.977281    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ef33f4-edc2-4996-9a67-9e369e1d98e9-kube-api-access-t5bs7" (OuterVolumeSpecName: "kube-api-access-t5bs7") pod "33ef33f4-edc2-4996-9a67-9e369e1d98e9" (UID: "33ef33f4-edc2-4996-9a67-9e369e1d98e9"). InnerVolumeSpecName "kube-api-access-t5bs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:06:56.053701 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:56.053675    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:56.075835 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:56.075803    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5bs7\" (UniqueName: \"kubernetes.io/projected/33ef33f4-edc2-4996-9a67-9e369e1d98e9-kube-api-access-t5bs7\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:06:56.075835 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:56.075836    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ef33f4-edc2-4996-9a67-9e369e1d98e9-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:06:56.115337 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:56.115311    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:56.255426 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:56.255332    2551 generic.go:356] "Generic (PLEG): container finished" podID="33ef33f4-edc2-4996-9a67-9e369e1d98e9" containerID="fc30dcd3d06b38595780b3d95946744992a503352c55a2b960f0469bb045c56d" exitCode=0
Aug 01 00:06:56.255426 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:56.255380    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"
Aug 01 00:06:56.255609 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:56.255421    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d" event={"ID":"33ef33f4-edc2-4996-9a67-9e369e1d98e9","Type":"ContainerDied","Data":"fc30dcd3d06b38595780b3d95946744992a503352c55a2b960f0469bb045c56d"}
Aug 01 00:06:56.255609 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:56.255479    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d" event={"ID":"33ef33f4-edc2-4996-9a67-9e369e1d98e9","Type":"ContainerDied","Data":"e13af1ecddd1e7dae6aa20ec5cc64870e563348a732071506774a52e329f6df5"}
Aug 01 00:06:56.255609 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:56.255504    2551 scope.go:117] "RemoveContainer" containerID="fc30dcd3d06b38595780b3d95946744992a503352c55a2b960f0469bb045c56d"
Aug 01 00:06:56.266671 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:56.266655    2551 scope.go:117] "RemoveContainer" containerID="fc30dcd3d06b38595780b3d95946744992a503352c55a2b960f0469bb045c56d"
Aug 01 00:06:56.266904 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:06:56.266888    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc30dcd3d06b38595780b3d95946744992a503352c55a2b960f0469bb045c56d\": container with ID starting with fc30dcd3d06b38595780b3d95946744992a503352c55a2b960f0469bb045c56d not found: ID does not exist" containerID="fc30dcd3d06b38595780b3d95946744992a503352c55a2b960f0469bb045c56d"
Aug 01 00:06:56.266994 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:56.266912    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc30dcd3d06b38595780b3d95946744992a503352c55a2b960f0469bb045c56d"} err="failed to get container status \"fc30dcd3d06b38595780b3d95946744992a503352c55a2b960f0469bb045c56d\": rpc error: code = NotFound desc = could not find container \"fc30dcd3d06b38595780b3d95946744992a503352c55a2b960f0469bb045c56d\": container with ID starting with fc30dcd3d06b38595780b3d95946744992a503352c55a2b960f0469bb045c56d not found: ID does not exist"
Aug 01 00:06:56.284198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:56.284174    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"]
Aug 01 00:06:56.297967 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:56.297947    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9h5r8d"]
Aug 01 00:06:57.946219 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:57.946193    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33ef33f4-edc2-4996-9a67-9e369e1d98e9" path="/var/lib/kubelet/pods/33ef33f4-edc2-4996-9a67-9e369e1d98e9/volumes"
Aug 01 00:06:59.090161 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.090123    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:59.256609 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.256563    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"]
Aug 01 00:06:59.257317 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.257289    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42d4d01b-82c0-44c4-81c5-89dd713ec82e" containerName="main"
Aug 01 00:06:59.257317 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.257317    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d4d01b-82c0-44c4-81c5-89dd713ec82e" containerName="main"
Aug 01 00:06:59.257463 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.257331    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ef33f4-edc2-4996-9a67-9e369e1d98e9" containerName="main"
Aug 01 00:06:59.257463 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.257341    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ef33f4-edc2-4996-9a67-9e369e1d98e9" containerName="main"
Aug 01 00:06:59.257463 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.257386    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c580659a-f373-43d3-be3b-e28e4736ee78" containerName="main"
Aug 01 00:06:59.257463 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.257394    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c580659a-f373-43d3-be3b-e28e4736ee78" containerName="main"
Aug 01 00:06:59.257648 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.257524    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="42d4d01b-82c0-44c4-81c5-89dd713ec82e" containerName="main"
Aug 01 00:06:59.257648 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.257536    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c580659a-f373-43d3-be3b-e28e4736ee78" containerName="main"
Aug 01 00:06:59.257648 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.257552    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ef33f4-edc2-4996-9a67-9e369e1d98e9" containerName="main"
Aug 01 00:06:59.262403 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.262380    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"
Aug 01 00:06:59.265125 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.265100    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-cnfsn\""
Aug 01 00:06:59.265281 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.265182    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Aug 01 00:06:59.265512 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.265491    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Aug 01 00:06:59.265672 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.265565    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Aug 01 00:06:59.266383 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.265662    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-9rnhm\""
Aug 01 00:06:59.268051 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.268030    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"]
Aug 01 00:06:59.280848 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.280821    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:59.403744 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.403653    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d59xf\" (UniqueName: \"kubernetes.io/projected/817a3391-2cec-4241-a55f-8e64d995bcea-kube-api-access-d59xf\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh\" (UID: \"817a3391-2cec-4241-a55f-8e64d995bcea\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"
Aug 01 00:06:59.403744 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.403704    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/817a3391-2cec-4241-a55f-8e64d995bcea-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh\" (UID: \"817a3391-2cec-4241-a55f-8e64d995bcea\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"
Aug 01 00:06:59.505098 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.505059    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d59xf\" (UniqueName: \"kubernetes.io/projected/817a3391-2cec-4241-a55f-8e64d995bcea-kube-api-access-d59xf\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh\" (UID: \"817a3391-2cec-4241-a55f-8e64d995bcea\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"
Aug 01 00:06:59.505278 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.505108    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/817a3391-2cec-4241-a55f-8e64d995bcea-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh\" (UID: \"817a3391-2cec-4241-a55f-8e64d995bcea\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"
Aug 01 00:06:59.507554 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.507531    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/817a3391-2cec-4241-a55f-8e64d995bcea-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh\" (UID: \"817a3391-2cec-4241-a55f-8e64d995bcea\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"
Aug 01 00:06:59.512538 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.512514    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d59xf\" (UniqueName: \"kubernetes.io/projected/817a3391-2cec-4241-a55f-8e64d995bcea-kube-api-access-d59xf\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh\" (UID: \"817a3391-2cec-4241-a55f-8e64d995bcea\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"
Aug 01 00:06:59.573704 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.573673    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"
Aug 01 00:06:59.699926 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:06:59.699888    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"]
Aug 01 00:06:59.702521 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:06:59.702490    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817a3391_2cec_4241_a55f_8e64d995bcea.slice/crio-584e1d55f572420d70af45274e70ee6947b076a6e2beb5d1e044fc1b9460fd7a WatchSource:0}: Error finding container 584e1d55f572420d70af45274e70ee6947b076a6e2beb5d1e044fc1b9460fd7a: Status 404 returned error can't find the container with id 584e1d55f572420d70af45274e70ee6947b076a6e2beb5d1e044fc1b9460fd7a
Aug 01 00:07:00.274610 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:00.274573    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh" event={"ID":"817a3391-2cec-4241-a55f-8e64d995bcea","Type":"ContainerStarted","Data":"fa46bc0c333016b8ce938583bf42c6c4087f732d04738d00d57c80f668b36bfb"}
Aug 01 00:07:00.274995 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:00.274626    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh" event={"ID":"817a3391-2cec-4241-a55f-8e64d995bcea","Type":"ContainerStarted","Data":"584e1d55f572420d70af45274e70ee6947b076a6e2beb5d1e044fc1b9460fd7a"}
Aug 01 00:07:00.274995 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:00.274657    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"
Aug 01 00:07:00.291785 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:00.291735    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh" podStartSLOduration=1.291719833 podStartE2EDuration="1.291719833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:07:00.290701177 +0000 UTC m=+1262.984748607" watchObservedRunningTime="2026-08-01 00:07:00.291719833 +0000 UTC m=+1262.985767248"
Aug 01 00:07:00.654121 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:00.654043    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:00.752996 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:00.752944    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:06.629967 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:06.629908    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:12.139593 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:12.139560    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:12.191582 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:12.191552    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:20.270257 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:20.270228    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:31.281370 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:31.281334    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"
Aug 01 00:07:35.441754 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:35.441715    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:35.674934 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:35.674894    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:41.871958 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:07:41.871898    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:09.303361 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:08:09.303323    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:09.402264 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:08:09.402230    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:29.232771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:08:29.232693    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:29.327808 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:08:29.327766    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:35.466240 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:08:35.466203    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:40.940337 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:08:40.940305    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:41.045042 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:08:41.045011    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:44.413614 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:08:44.413580    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:44.635355 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:08:44.635322    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:44.731591 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:08:44.731554    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:48.112872 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:08:48.112832    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:48.214553 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:08:48.214518    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:08:51.197213 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:08:51.197180    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:12.913499 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:12.913462    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"]
Aug 01 00:09:12.925914 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:12.925882    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:12.992598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:12.992569    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:21.281060 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:21.281017    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh" podUID="817a3391-2cec-4241-a55f-8e64d995bcea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:09:27.933124 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:27.933071    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh" podUID="817a3391-2cec-4241-a55f-8e64d995bcea" containerName="main" containerID="cri-o://fa46bc0c333016b8ce938583bf42c6c4087f732d04738d00d57c80f668b36bfb" gracePeriod=45
Aug 01 00:09:28.179237 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.179212    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"
Aug 01 00:09:28.240644 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.240613    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d59xf\" (UniqueName: \"kubernetes.io/projected/817a3391-2cec-4241-a55f-8e64d995bcea-kube-api-access-d59xf\") pod \"817a3391-2cec-4241-a55f-8e64d995bcea\" (UID: \"817a3391-2cec-4241-a55f-8e64d995bcea\") "
Aug 01 00:09:28.240804 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.240653    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/817a3391-2cec-4241-a55f-8e64d995bcea-tls-certs\") pod \"817a3391-2cec-4241-a55f-8e64d995bcea\" (UID: \"817a3391-2cec-4241-a55f-8e64d995bcea\") "
Aug 01 00:09:28.242888 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.242855    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817a3391-2cec-4241-a55f-8e64d995bcea-kube-api-access-d59xf" (OuterVolumeSpecName: "kube-api-access-d59xf") pod "817a3391-2cec-4241-a55f-8e64d995bcea" (UID: "817a3391-2cec-4241-a55f-8e64d995bcea"). InnerVolumeSpecName "kube-api-access-d59xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:09:28.243014 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.242898    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/817a3391-2cec-4241-a55f-8e64d995bcea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "817a3391-2cec-4241-a55f-8e64d995bcea" (UID: "817a3391-2cec-4241-a55f-8e64d995bcea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:09:28.342026 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.341995    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d59xf\" (UniqueName: \"kubernetes.io/projected/817a3391-2cec-4241-a55f-8e64d995bcea-kube-api-access-d59xf\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:09:28.342026 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.342020    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/817a3391-2cec-4241-a55f-8e64d995bcea-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:09:28.862612 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.862576    2551 generic.go:356] "Generic (PLEG): container finished" podID="817a3391-2cec-4241-a55f-8e64d995bcea" containerID="fa46bc0c333016b8ce938583bf42c6c4087f732d04738d00d57c80f668b36bfb" exitCode=0
Aug 01 00:09:28.862809 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.862618    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh" event={"ID":"817a3391-2cec-4241-a55f-8e64d995bcea","Type":"ContainerDied","Data":"fa46bc0c333016b8ce938583bf42c6c4087f732d04738d00d57c80f668b36bfb"}
Aug 01 00:09:28.862809 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.862626    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"
Aug 01 00:09:28.862809 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.862668    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh" event={"ID":"817a3391-2cec-4241-a55f-8e64d995bcea","Type":"ContainerDied","Data":"584e1d55f572420d70af45274e70ee6947b076a6e2beb5d1e044fc1b9460fd7a"}
Aug 01 00:09:28.862809 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.862690    2551 scope.go:117] "RemoveContainer" containerID="fa46bc0c333016b8ce938583bf42c6c4087f732d04738d00d57c80f668b36bfb"
Aug 01 00:09:28.871257 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.871238    2551 scope.go:117] "RemoveContainer" containerID="fa46bc0c333016b8ce938583bf42c6c4087f732d04738d00d57c80f668b36bfb"
Aug 01 00:09:28.871497 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:09:28.871476    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa46bc0c333016b8ce938583bf42c6c4087f732d04738d00d57c80f668b36bfb\": container with ID starting with fa46bc0c333016b8ce938583bf42c6c4087f732d04738d00d57c80f668b36bfb not found: ID does not exist" containerID="fa46bc0c333016b8ce938583bf42c6c4087f732d04738d00d57c80f668b36bfb"
Aug 01 00:09:28.871544 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.871505    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa46bc0c333016b8ce938583bf42c6c4087f732d04738d00d57c80f668b36bfb"} err="failed to get container status \"fa46bc0c333016b8ce938583bf42c6c4087f732d04738d00d57c80f668b36bfb\": rpc error: code = NotFound desc = could not find container \"fa46bc0c333016b8ce938583bf42c6c4087f732d04738d00d57c80f668b36bfb\": container with ID starting with fa46bc0c333016b8ce938583bf42c6c4087f732d04738d00d57c80f668b36bfb not found: ID does not exist"
Aug 01 00:09:28.886381 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.886319    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"]
Aug 01 00:09:28.889447 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:28.889426    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-4z6sh"]
Aug 01 00:09:29.948100 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:29.948052    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="817a3391-2cec-4241-a55f-8e64d995bcea" path="/var/lib/kubelet/pods/817a3391-2cec-4241-a55f-8e64d995bcea/volumes"
Aug 01 00:09:32.977658 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:32.977610    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:33.075769 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:33.075731    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:33.176418 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:33.176389    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:38.846897 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:38.846857    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:39.569983 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.569953    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"]
Aug 01 00:09:39.570249 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.570237    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817a3391-2cec-4241-a55f-8e64d995bcea" containerName="main"
Aug 01 00:09:39.570295 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.570250    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="817a3391-2cec-4241-a55f-8e64d995bcea" containerName="main"
Aug 01 00:09:39.570343 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.570316    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="817a3391-2cec-4241-a55f-8e64d995bcea" containerName="main"
Aug 01 00:09:39.574459 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.574440    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.579033 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.579005    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Aug 01 00:09:39.579033 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.579022    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Aug 01 00:09:39.579194 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.579032    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-w2ft2\""
Aug 01 00:09:39.579194 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.579061    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Aug 01 00:09:39.586241 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.586218    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"]
Aug 01 00:09:39.590248 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.590228    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:39.636959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.636935    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-model-cache\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.637096 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.636974    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-home\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.637096 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.637082    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.637183 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.637111    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133b3d42-ffd1-48c0-84ac-252d0201d59f-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.637183 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.637127    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vxmp\" (UniqueName: \"kubernetes.io/projected/133b3d42-ffd1-48c0-84ac-252d0201d59f-kube-api-access-4vxmp\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.637183 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.637143    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-dshm\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.738329 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.738297    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-model-cache\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.738490 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.738343    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-home\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.738490 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.738425    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.738490 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.738456    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133b3d42-ffd1-48c0-84ac-252d0201d59f-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.738490 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.738479    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vxmp\" (UniqueName: \"kubernetes.io/projected/133b3d42-ffd1-48c0-84ac-252d0201d59f-kube-api-access-4vxmp\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.738679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.738504    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-dshm\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.738802 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.738768    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-model-cache\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.738955 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.738810    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-home\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.738955 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.738850    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.740730 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.740711    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-dshm\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.740907 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.740890    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133b3d42-ffd1-48c0-84ac-252d0201d59f-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.745181 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.745161    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vxmp\" (UniqueName: \"kubernetes.io/projected/133b3d42-ffd1-48c0-84ac-252d0201d59f-kube-api-access-4vxmp\") pod \"canary-v1-kserve-65d58ccc9-r5wq8\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:39.884044 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:39.883953    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:40.027065 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:40.027033    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"]
Aug 01 00:09:40.029971 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:09:40.029942    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133b3d42_ffd1_48c0_84ac_252d0201d59f.slice/crio-ab2af80178c4ed321e3b1394ca3a2f7ab027136d7da2ebf2d5d2a242e586b18b WatchSource:0}: Error finding container ab2af80178c4ed321e3b1394ca3a2f7ab027136d7da2ebf2d5d2a242e586b18b: Status 404 returned error can't find the container with id ab2af80178c4ed321e3b1394ca3a2f7ab027136d7da2ebf2d5d2a242e586b18b
Aug 01 00:09:40.032053 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:40.032035    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:09:40.914353 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:40.914309    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8" event={"ID":"133b3d42-ffd1-48c0-84ac-252d0201d59f","Type":"ContainerStarted","Data":"ab2af80178c4ed321e3b1394ca3a2f7ab027136d7da2ebf2d5d2a242e586b18b"}
Aug 01 00:09:41.920392 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:41.920347    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8" event={"ID":"133b3d42-ffd1-48c0-84ac-252d0201d59f","Type":"ContainerStarted","Data":"8e2af03e7e7f95aa110e9e30da92e85a779d6d5547a03209066c0deda0927469"}
Aug 01 00:09:41.936461 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:41.936406    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8" podStartSLOduration=1.7319990920000001 podStartE2EDuration="2.9363868s" totalImagesPullingTime="1.204387708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:09:41.935385449 +0000 UTC m=+1424.629432864" watchObservedRunningTime="2026-08-01 00:09:41.9363868 +0000 UTC m=+1424.630434217"
Aug 01 00:09:42.884590 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:42.884542    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:44.466197 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:44.466143    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:44.567477 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:44.567434    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:49.884804 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:49.884768    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:49.897544 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:49.897517    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:49.964742 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:49.964715    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:09:57.953712 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:57.953683    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"]
Aug 01 00:09:57.958325 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:57.958306    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:57.960261 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:57.960245    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Aug 01 00:09:57.964291 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:57.964263    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"]
Aug 01 00:09:58.051709 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.051677    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:58.097877 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.097850    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-dshm\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.098012 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.097881    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.098012 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.097900    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nwhc\" (UniqueName: \"kubernetes.io/projected/8d8999de-60b4-4c8d-98ee-6b2863333c84-kube-api-access-7nwhc\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.098012 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.097938    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d8999de-60b4-4c8d-98ee-6b2863333c84-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.098012 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.097991    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-home\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.098185 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.098078    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.199251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.199158    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-dshm\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.199251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.199197    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.199251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.199220    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nwhc\" (UniqueName: \"kubernetes.io/projected/8d8999de-60b4-4c8d-98ee-6b2863333c84-kube-api-access-7nwhc\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.199527 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.199326    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d8999de-60b4-4c8d-98ee-6b2863333c84-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.199527 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.199357    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-home\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.199629 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.199577    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.199680 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.199662    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.199838 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.199817    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-home\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.200033 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.200011    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.201519 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.201493    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-dshm\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.201779 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.201763    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d8999de-60b4-4c8d-98ee-6b2863333c84-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.206640 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.206621    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nwhc\" (UniqueName: \"kubernetes.io/projected/8d8999de-60b4-4c8d-98ee-6b2863333c84-kube-api-access-7nwhc\") pod \"canary-v2-kserve-7f65dbbccc-7p6dr\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.269574 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.269543    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:09:58.397721 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.397696    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"]
Aug 01 00:09:58.400352 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:09:58.400305    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8999de_60b4_4c8d_98ee_6b2863333c84.slice/crio-5a5d6c87e4e25e32fe2eb3b1e800d4bd04e75f755c831ea9bf4f92aa2dec678a WatchSource:0}: Error finding container 5a5d6c87e4e25e32fe2eb3b1e800d4bd04e75f755c831ea9bf4f92aa2dec678a: Status 404 returned error can't find the container with id 5a5d6c87e4e25e32fe2eb3b1e800d4bd04e75f755c831ea9bf4f92aa2dec678a
Aug 01 00:09:58.997327 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.997279    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr" event={"ID":"8d8999de-60b4-4c8d-98ee-6b2863333c84","Type":"ContainerStarted","Data":"a68451dcc518b26c23a6df1a8020ab21660d698bdf80a68104edfc7fb1611a1f"}
Aug 01 00:09:58.997806 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:58.997385    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr" event={"ID":"8d8999de-60b4-4c8d-98ee-6b2863333c84","Type":"ContainerStarted","Data":"5a5d6c87e4e25e32fe2eb3b1e800d4bd04e75f755c831ea9bf4f92aa2dec678a"}
Aug 01 00:09:59.017636 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:59.017563    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr" podStartSLOduration=2.017542035 podStartE2EDuration="2.017542035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:09:59.014929263 +0000 UTC m=+1441.708976671" watchObservedRunningTime="2026-08-01 00:09:59.017542035 +0000 UTC m=+1441.711589452"
Aug 01 00:09:59.270097 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:09:59.270011    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:10:00.299642 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:00.299599    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:08.270708 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:08.270675    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:10:08.283183 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:08.283157    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:10:09.053605 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:09.053565    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:10:11.547385 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:11.547341    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:13.784433 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:13.784393    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:13.871316 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:13.871281    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:31.287604 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:31.287570    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:33.934904 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:33.934871    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:34.032533 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:34.032482    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:41.280085 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:41.280045    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:45.263910 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:45.263873    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:45.365070 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:45.365036    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:50.391036 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.391001    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"]
Aug 01 00:10:50.394563 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.394544    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.397379 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.397347    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Aug 01 00:10:50.397511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.397394    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Aug 01 00:10:50.397511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.397415    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Aug 01 00:10:50.397511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.397398    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-9cc8q\""
Aug 01 00:10:50.397511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.397396    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:10:50.398292 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.398276    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Aug 01 00:10:50.407114 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.407090    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"]
Aug 01 00:10:50.422772 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.422742    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:50.545028 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.544990    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-dshm\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.545195 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.545068    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-model-cache\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.545195 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.545116    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7l5j\" (UniqueName: \"kubernetes.io/projected/83746e3e-0540-44e5-adf5-878e7c4bc460-kube-api-access-s7l5j\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.545195 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.545145    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83746e3e-0540-44e5-adf5-878e7c4bc460-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.545195 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.545169    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.545388 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.545239    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.545388 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.545289    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83746e3e-0540-44e5-adf5-878e7c4bc460-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.545388 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.545333    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-home\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.646656 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.646571    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-dshm\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.646656 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.646632    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-model-cache\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.646866 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.646671    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7l5j\" (UniqueName: \"kubernetes.io/projected/83746e3e-0540-44e5-adf5-878e7c4bc460-kube-api-access-s7l5j\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.646866 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.646697    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83746e3e-0540-44e5-adf5-878e7c4bc460-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.646866 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.646724    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.646866 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.646756    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.646866 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.646783    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83746e3e-0540-44e5-adf5-878e7c4bc460-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.646866 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.646817    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-home\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.647237 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.647106    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-model-cache\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.647237 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.647169    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.647421 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.647389    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-home\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.647677 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.647655    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.647806 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.647784    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83746e3e-0540-44e5-adf5-878e7c4bc460-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.648974 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.648949    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-dshm\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.649315 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.649290    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83746e3e-0540-44e5-adf5-878e7c4bc460-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.657172 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.657143    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7l5j\" (UniqueName: \"kubernetes.io/projected/83746e3e-0540-44e5-adf5-878e7c4bc460-kube-api-access-s7l5j\") pod \"tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.662233 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.662210    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"]
Aug 01 00:10:50.666961 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.666942    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"
Aug 01 00:10:50.669122 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.669102    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-4r522\""
Aug 01 00:10:50.670895 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.670863    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:50.674489 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.674427    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"]
Aug 01 00:10:50.704981 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.704952    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:10:50.748103 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.748072    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fa8554-8716-45d1-8028-aeddeb6a3da8-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j\" (UID: \"09fa8554-8716-45d1-8028-aeddeb6a3da8\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"
Aug 01 00:10:50.748267 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.748112    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnltr\" (UniqueName: \"kubernetes.io/projected/09fa8554-8716-45d1-8028-aeddeb6a3da8-kube-api-access-nnltr\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j\" (UID: \"09fa8554-8716-45d1-8028-aeddeb6a3da8\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"
Aug 01 00:10:50.769622 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.769588    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:50.839508 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.839484    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"]
Aug 01 00:10:50.841501 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:10:50.841472    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83746e3e_0540_44e5_adf5_878e7c4bc460.slice/crio-a94b489b9aabadc8c4cd016d646d4df86d7d2cd386dc7114790ef1ccc87cf7cc WatchSource:0}: Error finding container a94b489b9aabadc8c4cd016d646d4df86d7d2cd386dc7114790ef1ccc87cf7cc: Status 404 returned error can't find the container with id a94b489b9aabadc8c4cd016d646d4df86d7d2cd386dc7114790ef1ccc87cf7cc
Aug 01 00:10:50.848691 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.848623    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fa8554-8716-45d1-8028-aeddeb6a3da8-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j\" (UID: \"09fa8554-8716-45d1-8028-aeddeb6a3da8\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"
Aug 01 00:10:50.848691 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.848654    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnltr\" (UniqueName: \"kubernetes.io/projected/09fa8554-8716-45d1-8028-aeddeb6a3da8-kube-api-access-nnltr\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j\" (UID: \"09fa8554-8716-45d1-8028-aeddeb6a3da8\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"
Aug 01 00:10:50.851026 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.851002    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fa8554-8716-45d1-8028-aeddeb6a3da8-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j\" (UID: \"09fa8554-8716-45d1-8028-aeddeb6a3da8\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"
Aug 01 00:10:50.857335 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.857293    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnltr\" (UniqueName: \"kubernetes.io/projected/09fa8554-8716-45d1-8028-aeddeb6a3da8-kube-api-access-nnltr\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j\" (UID: \"09fa8554-8716-45d1-8028-aeddeb6a3da8\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"
Aug 01 00:10:50.979294 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:50.979263    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"
Aug 01 00:10:51.115414 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:51.115384    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"]
Aug 01 00:10:51.117533 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:10:51.117506    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fa8554_8716_45d1_8028_aeddeb6a3da8.slice/crio-e2054692a0e0f15f0fd44818046512020f0c388f8d5accdeecc7e993f3826e55 WatchSource:0}: Error finding container e2054692a0e0f15f0fd44818046512020f0c388f8d5accdeecc7e993f3826e55: Status 404 returned error can't find the container with id e2054692a0e0f15f0fd44818046512020f0c388f8d5accdeecc7e993f3826e55
Aug 01 00:10:51.211021 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:51.210984    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j" event={"ID":"09fa8554-8716-45d1-8028-aeddeb6a3da8","Type":"ContainerStarted","Data":"34e164391de3e11dcccb44db22d40ad954fed5f12f33f44acd5a1417b52f9df4"}
Aug 01 00:10:51.211202 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:51.211050    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"
Aug 01 00:10:51.211202 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:51.211077    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j" event={"ID":"09fa8554-8716-45d1-8028-aeddeb6a3da8","Type":"ContainerStarted","Data":"e2054692a0e0f15f0fd44818046512020f0c388f8d5accdeecc7e993f3826e55"}
Aug 01 00:10:51.212626 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:51.212596    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz" event={"ID":"83746e3e-0540-44e5-adf5-878e7c4bc460","Type":"ContainerStarted","Data":"ba2480206bfa41d7ae4c202de9d70fcbd98478c29e69df7078d550d68fcbc95f"}
Aug 01 00:10:51.212752 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:51.212655    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz" event={"ID":"83746e3e-0540-44e5-adf5-878e7c4bc460","Type":"ContainerStarted","Data":"a94b489b9aabadc8c4cd016d646d4df86d7d2cd386dc7114790ef1ccc87cf7cc"}
Aug 01 00:10:51.226378 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:51.226332    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j" podStartSLOduration=1.226317906 podStartE2EDuration="1.226317906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:10:51.224885803 +0000 UTC m=+1493.918933230" watchObservedRunningTime="2026-08-01 00:10:51.226317906 +0000 UTC m=+1493.920365321"
Aug 01 00:10:53.225650 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:53.225612    2551 generic.go:356] "Generic (PLEG): container finished" podID="83746e3e-0540-44e5-adf5-878e7c4bc460" containerID="ba2480206bfa41d7ae4c202de9d70fcbd98478c29e69df7078d550d68fcbc95f" exitCode=0
Aug 01 00:10:53.226043 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:53.225696    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz" event={"ID":"83746e3e-0540-44e5-adf5-878e7c4bc460","Type":"ContainerDied","Data":"ba2480206bfa41d7ae4c202de9d70fcbd98478c29e69df7078d550d68fcbc95f"}
Aug 01 00:10:54.231676 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:54.231639    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz" event={"ID":"83746e3e-0540-44e5-adf5-878e7c4bc460","Type":"ContainerStarted","Data":"747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a"}
Aug 01 00:10:54.253554 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:54.253503    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz" podStartSLOduration=2.100210441 podStartE2EDuration="4.253487328s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.153276887s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:10:54.251413021 +0000 UTC m=+1496.945460444" watchObservedRunningTime="2026-08-01 00:10:54.253487328 +0000 UTC m=+1496.947534788"
Aug 01 00:10:54.705470 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:10:54.705356    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:11:00.705514 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:00.705479    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:11:00.718107 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:00.718079    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:11:01.274342 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:01.274316    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:11:06.172405 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:06.172375    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:22.221550 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:22.221468    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"
Aug 01 00:11:26.962558 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:26.962521    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:26.970856 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:26.970828    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"]
Aug 01 00:11:26.978992 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:26.978966    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"]
Aug 01 00:11:26.987933 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:11:26.987886    2551 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:11:26.987933 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:11:26.987933 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:11:26.987933 ip-10-0-131-136 kubenswrapper[2551]:  > containerID="747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a" cmd=["/bin/sleep","15"]
Aug 01 00:11:26.988213 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:26.987961    2551 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz" message="" err=<
Aug 01 00:11:26.988213 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:11:26.988213 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:11:26.988213 ip-10-0-131-136 kubenswrapper[2551]:  >
Aug 01 00:11:26.988213 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:11:26.987981    2551 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:11:26.988213 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:11:26.988213 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:11:26.988213 ip-10-0-131-136 kubenswrapper[2551]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz" podUID="83746e3e-0540-44e5-adf5-878e7c4bc460" containerName="main" containerID="cri-o://747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a"
Aug 01 00:11:26.988213 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:26.988021    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz" podUID="83746e3e-0540-44e5-adf5-878e7c4bc460" containerName="main" containerID="cri-o://747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a" gracePeriod=60
Aug 01 00:11:27.062458 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.062422    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:27.357290 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.357254    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:11:27.381102 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.381054    2551 generic.go:356] "Generic (PLEG): container finished" podID="83746e3e-0540-44e5-adf5-878e7c4bc460" containerID="747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a" exitCode=0
Aug 01 00:11:27.381281 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.381156    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz" event={"ID":"83746e3e-0540-44e5-adf5-878e7c4bc460","Type":"ContainerDied","Data":"747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a"}
Aug 01 00:11:27.381281 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.381240    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz" event={"ID":"83746e3e-0540-44e5-adf5-878e7c4bc460","Type":"ContainerDied","Data":"a94b489b9aabadc8c4cd016d646d4df86d7d2cd386dc7114790ef1ccc87cf7cc"}
Aug 01 00:11:27.381281 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.381269    2551 scope.go:117] "RemoveContainer" containerID="747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a"
Aug 01 00:11:27.381281 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.381269    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"
Aug 01 00:11:27.393686 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.393661    2551 scope.go:117] "RemoveContainer" containerID="ba2480206bfa41d7ae4c202de9d70fcbd98478c29e69df7078d550d68fcbc95f"
Aug 01 00:11:27.404192 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.404169    2551 scope.go:117] "RemoveContainer" containerID="747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a"
Aug 01 00:11:27.404494 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:11:27.404463    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a\": container with ID starting with 747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a not found: ID does not exist" containerID="747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a"
Aug 01 00:11:27.404616 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.404505    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a"} err="failed to get container status \"747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a\": rpc error: code = NotFound desc = could not find container \"747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a\": container with ID starting with 747bce668c63b5a8caa4d0459629c84ba10619f7fa8621c72032e17a2a505a2a not found: ID does not exist"
Aug 01 00:11:27.404616 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.404530    2551 scope.go:117] "RemoveContainer" containerID="ba2480206bfa41d7ae4c202de9d70fcbd98478c29e69df7078d550d68fcbc95f"
Aug 01 00:11:27.404820 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:11:27.404799    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2480206bfa41d7ae4c202de9d70fcbd98478c29e69df7078d550d68fcbc95f\": container with ID starting with ba2480206bfa41d7ae4c202de9d70fcbd98478c29e69df7078d550d68fcbc95f not found: ID does not exist" containerID="ba2480206bfa41d7ae4c202de9d70fcbd98478c29e69df7078d550d68fcbc95f"
Aug 01 00:11:27.404887 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.404835    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2480206bfa41d7ae4c202de9d70fcbd98478c29e69df7078d550d68fcbc95f"} err="failed to get container status \"ba2480206bfa41d7ae4c202de9d70fcbd98478c29e69df7078d550d68fcbc95f\": rpc error: code = NotFound desc = could not find container \"ba2480206bfa41d7ae4c202de9d70fcbd98478c29e69df7078d550d68fcbc95f\": container with ID starting with ba2480206bfa41d7ae4c202de9d70fcbd98478c29e69df7078d550d68fcbc95f not found: ID does not exist"
Aug 01 00:11:27.481122 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.481083    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7l5j\" (UniqueName: \"kubernetes.io/projected/83746e3e-0540-44e5-adf5-878e7c4bc460-kube-api-access-s7l5j\") pod \"83746e3e-0540-44e5-adf5-878e7c4bc460\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") "
Aug 01 00:11:27.481314 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.481143    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-home\") pod \"83746e3e-0540-44e5-adf5-878e7c4bc460\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") "
Aug 01 00:11:27.481314 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.481199    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-tmp-dir\") pod \"83746e3e-0540-44e5-adf5-878e7c4bc460\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") "
Aug 01 00:11:27.481314 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.481226    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83746e3e-0540-44e5-adf5-878e7c4bc460-cabundle-cert\") pod \"83746e3e-0540-44e5-adf5-878e7c4bc460\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") "
Aug 01 00:11:27.481314 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.481268    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-dshm\") pod \"83746e3e-0540-44e5-adf5-878e7c4bc460\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") "
Aug 01 00:11:27.481314 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.481294    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-kserve-provision-location\") pod \"83746e3e-0540-44e5-adf5-878e7c4bc460\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") "
Aug 01 00:11:27.481624 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.481344    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83746e3e-0540-44e5-adf5-878e7c4bc460-tls-certs\") pod \"83746e3e-0540-44e5-adf5-878e7c4bc460\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") "
Aug 01 00:11:27.481624 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.481381    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-home" (OuterVolumeSpecName: "home") pod "83746e3e-0540-44e5-adf5-878e7c4bc460" (UID: "83746e3e-0540-44e5-adf5-878e7c4bc460"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:27.481624 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.481392    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-model-cache\") pod \"83746e3e-0540-44e5-adf5-878e7c4bc460\" (UID: \"83746e3e-0540-44e5-adf5-878e7c4bc460\") "
Aug 01 00:11:27.481624 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.481549    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-model-cache" (OuterVolumeSpecName: "model-cache") pod "83746e3e-0540-44e5-adf5-878e7c4bc460" (UID: "83746e3e-0540-44e5-adf5-878e7c4bc460"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:27.481833 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.481716    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.481833 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.481737    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.482027 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.481897    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83746e3e-0540-44e5-adf5-878e7c4bc460" (UID: "83746e3e-0540-44e5-adf5-878e7c4bc460"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:27.483118 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.482149    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83746e3e-0540-44e5-adf5-878e7c4bc460-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83746e3e-0540-44e5-adf5-878e7c4bc460" (UID: "83746e3e-0540-44e5-adf5-878e7c4bc460"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:11:27.485240 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.485041    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83746e3e-0540-44e5-adf5-878e7c4bc460-kube-api-access-s7l5j" (OuterVolumeSpecName: "kube-api-access-s7l5j") pod "83746e3e-0540-44e5-adf5-878e7c4bc460" (UID: "83746e3e-0540-44e5-adf5-878e7c4bc460"). InnerVolumeSpecName "kube-api-access-s7l5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:11:27.488208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.485356    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83746e3e-0540-44e5-adf5-878e7c4bc460-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83746e3e-0540-44e5-adf5-878e7c4bc460" (UID: "83746e3e-0540-44e5-adf5-878e7c4bc460"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:11:27.488208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.486625    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-dshm" (OuterVolumeSpecName: "dshm") pod "83746e3e-0540-44e5-adf5-878e7c4bc460" (UID: "83746e3e-0540-44e5-adf5-878e7c4bc460"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:27.514898 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.514817    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83746e3e-0540-44e5-adf5-878e7c4bc460" (UID: "83746e3e-0540-44e5-adf5-878e7c4bc460"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:27.582777 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.582739    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.582777 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.582777    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83746e3e-0540-44e5-adf5-878e7c4bc460-cabundle-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.583009 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.582790    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.583009 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.582803    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83746e3e-0540-44e5-adf5-878e7c4bc460-kserve-provision-location\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.583009 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.582815    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83746e3e-0540-44e5-adf5-878e7c4bc460-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.583009 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.582828    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7l5j\" (UniqueName: \"kubernetes.io/projected/83746e3e-0540-44e5-adf5-878e7c4bc460-kube-api-access-s7l5j\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:27.702656 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.702623    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"]
Aug 01 00:11:27.706510 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.706484    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz"]
Aug 01 00:11:27.947133 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:27.947060    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83746e3e-0540-44e5-adf5-878e7c4bc460" path="/var/lib/kubelet/pods/83746e3e-0540-44e5-adf5-878e7c4bc460/volumes"
Aug 01 00:11:28.265208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:28.265170    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6bbc5f5c69-zv4jz" podUID="83746e3e-0540-44e5-adf5-878e7c4bc460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Aug 01 00:11:32.221386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:32.221346    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j" podUID="09fa8554-8716-45d1-8028-aeddeb6a3da8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:11:34.103996 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:34.103943    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:34.199768 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:34.199734    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:34.646963 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:34.646908    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"]
Aug 01 00:11:34.658348 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:34.658313    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:38.597566 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:38.597528    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:39.030458 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:39.030419    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"]
Aug 01 00:11:39.077930 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:39.077890    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:39.758048 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:39.758016    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:41.992017 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:41.991974    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j" podUID="09fa8554-8716-45d1-8028-aeddeb6a3da8" containerName="main" containerID="cri-o://34e164391de3e11dcccb44db22d40ad954fed5f12f33f44acd5a1417b52f9df4" gracePeriod=45
Aug 01 00:11:42.236237 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.236211    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"
Aug 01 00:11:42.404897 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.404806    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnltr\" (UniqueName: \"kubernetes.io/projected/09fa8554-8716-45d1-8028-aeddeb6a3da8-kube-api-access-nnltr\") pod \"09fa8554-8716-45d1-8028-aeddeb6a3da8\" (UID: \"09fa8554-8716-45d1-8028-aeddeb6a3da8\") "
Aug 01 00:11:42.404897 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.404846    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fa8554-8716-45d1-8028-aeddeb6a3da8-tls-certs\") pod \"09fa8554-8716-45d1-8028-aeddeb6a3da8\" (UID: \"09fa8554-8716-45d1-8028-aeddeb6a3da8\") "
Aug 01 00:11:42.407010 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.406977    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fa8554-8716-45d1-8028-aeddeb6a3da8-kube-api-access-nnltr" (OuterVolumeSpecName: "kube-api-access-nnltr") pod "09fa8554-8716-45d1-8028-aeddeb6a3da8" (UID: "09fa8554-8716-45d1-8028-aeddeb6a3da8"). InnerVolumeSpecName "kube-api-access-nnltr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:11:42.407126 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.407040    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09fa8554-8716-45d1-8028-aeddeb6a3da8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09fa8554-8716-45d1-8028-aeddeb6a3da8" (UID: "09fa8554-8716-45d1-8028-aeddeb6a3da8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:11:42.449961 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.449913    2551 generic.go:356] "Generic (PLEG): container finished" podID="09fa8554-8716-45d1-8028-aeddeb6a3da8" containerID="34e164391de3e11dcccb44db22d40ad954fed5f12f33f44acd5a1417b52f9df4" exitCode=0
Aug 01 00:11:42.450073 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.449976    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"
Aug 01 00:11:42.450073 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.450014    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j" event={"ID":"09fa8554-8716-45d1-8028-aeddeb6a3da8","Type":"ContainerDied","Data":"34e164391de3e11dcccb44db22d40ad954fed5f12f33f44acd5a1417b52f9df4"}
Aug 01 00:11:42.450073 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.450067    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j" event={"ID":"09fa8554-8716-45d1-8028-aeddeb6a3da8","Type":"ContainerDied","Data":"e2054692a0e0f15f0fd44818046512020f0c388f8d5accdeecc7e993f3826e55"}
Aug 01 00:11:42.450176 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.450090    2551 scope.go:117] "RemoveContainer" containerID="34e164391de3e11dcccb44db22d40ad954fed5f12f33f44acd5a1417b52f9df4"
Aug 01 00:11:42.458356 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.458341    2551 scope.go:117] "RemoveContainer" containerID="34e164391de3e11dcccb44db22d40ad954fed5f12f33f44acd5a1417b52f9df4"
Aug 01 00:11:42.458585 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:11:42.458569    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34e164391de3e11dcccb44db22d40ad954fed5f12f33f44acd5a1417b52f9df4\": container with ID starting with 34e164391de3e11dcccb44db22d40ad954fed5f12f33f44acd5a1417b52f9df4 not found: ID does not exist" containerID="34e164391de3e11dcccb44db22d40ad954fed5f12f33f44acd5a1417b52f9df4"
Aug 01 00:11:42.458627 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.458589    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e164391de3e11dcccb44db22d40ad954fed5f12f33f44acd5a1417b52f9df4"} err="failed to get container status \"34e164391de3e11dcccb44db22d40ad954fed5f12f33f44acd5a1417b52f9df4\": rpc error: code = NotFound desc = could not find container \"34e164391de3e11dcccb44db22d40ad954fed5f12f33f44acd5a1417b52f9df4\": container with ID starting with 34e164391de3e11dcccb44db22d40ad954fed5f12f33f44acd5a1417b52f9df4 not found: ID does not exist"
Aug 01 00:11:42.469467 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.469440    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"]
Aug 01 00:11:42.473053 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.473032    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j"]
Aug 01 00:11:42.506524 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.506499    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnltr\" (UniqueName: \"kubernetes.io/projected/09fa8554-8716-45d1-8028-aeddeb6a3da8-kube-api-access-nnltr\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:42.506524 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:42.506520    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fa8554-8716-45d1-8028-aeddeb6a3da8-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:43.220778 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:43.220734    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9ghp8j" podUID="09fa8554-8716-45d1-8028-aeddeb6a3da8" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.45:9003\" within 1s: context deadline exceeded"
Aug 01 00:11:43.221552 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:11:43.221529    2551 logging.go:55] [core] [Channel #332 SubChannel #333]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.45:9003", ServerName: "10.133.0.45:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.45:9003: operation was canceled"
Aug 01 00:11:43.946197 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:43.946160    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09fa8554-8716-45d1-8028-aeddeb6a3da8" path="/var/lib/kubelet/pods/09fa8554-8716-45d1-8028-aeddeb6a3da8/volumes"
Aug 01 00:11:45.273932 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:45.273887    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:47.121305 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:47.121269    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:47.219786 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:47.219754    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:49.668010 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:49.667969    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8" podUID="133b3d42-ffd1-48c0-84ac-252d0201d59f" containerName="main" containerID="cri-o://8e2af03e7e7f95aa110e9e30da92e85a779d6d5547a03209066c0deda0927469" gracePeriod=45
Aug 01 00:11:49.914216 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:49.914190    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:11:50.068618 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.068588    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vxmp\" (UniqueName: \"kubernetes.io/projected/133b3d42-ffd1-48c0-84ac-252d0201d59f-kube-api-access-4vxmp\") pod \"133b3d42-ffd1-48c0-84ac-252d0201d59f\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") "
Aug 01 00:11:50.068831 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.068630    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-model-cache\") pod \"133b3d42-ffd1-48c0-84ac-252d0201d59f\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") "
Aug 01 00:11:50.068831 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.068729    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133b3d42-ffd1-48c0-84ac-252d0201d59f-tls-certs\") pod \"133b3d42-ffd1-48c0-84ac-252d0201d59f\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") "
Aug 01 00:11:50.068831 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.068755    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-dshm\") pod \"133b3d42-ffd1-48c0-84ac-252d0201d59f\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") "
Aug 01 00:11:50.068831 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.068784    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-tmp-dir\") pod \"133b3d42-ffd1-48c0-84ac-252d0201d59f\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") "
Aug 01 00:11:50.068831 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.068811    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-home\") pod \"133b3d42-ffd1-48c0-84ac-252d0201d59f\" (UID: \"133b3d42-ffd1-48c0-84ac-252d0201d59f\") "
Aug 01 00:11:50.069150 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.068938    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-model-cache" (OuterVolumeSpecName: "model-cache") pod "133b3d42-ffd1-48c0-84ac-252d0201d59f" (UID: "133b3d42-ffd1-48c0-84ac-252d0201d59f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:50.069150 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.069089    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "133b3d42-ffd1-48c0-84ac-252d0201d59f" (UID: "133b3d42-ffd1-48c0-84ac-252d0201d59f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:50.069254 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.069146    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-home" (OuterVolumeSpecName: "home") pod "133b3d42-ffd1-48c0-84ac-252d0201d59f" (UID: "133b3d42-ffd1-48c0-84ac-252d0201d59f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:50.069254 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.069159    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:50.070992 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.070961    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-dshm" (OuterVolumeSpecName: "dshm") pod "133b3d42-ffd1-48c0-84ac-252d0201d59f" (UID: "133b3d42-ffd1-48c0-84ac-252d0201d59f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:50.071116 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.071026    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133b3d42-ffd1-48c0-84ac-252d0201d59f-kube-api-access-4vxmp" (OuterVolumeSpecName: "kube-api-access-4vxmp") pod "133b3d42-ffd1-48c0-84ac-252d0201d59f" (UID: "133b3d42-ffd1-48c0-84ac-252d0201d59f"). InnerVolumeSpecName "kube-api-access-4vxmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:11:50.071116 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.071053    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/133b3d42-ffd1-48c0-84ac-252d0201d59f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "133b3d42-ffd1-48c0-84ac-252d0201d59f" (UID: "133b3d42-ffd1-48c0-84ac-252d0201d59f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:11:50.169679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.169637    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:50.169679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.169672    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:50.169679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.169685    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vxmp\" (UniqueName: \"kubernetes.io/projected/133b3d42-ffd1-48c0-84ac-252d0201d59f-kube-api-access-4vxmp\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:50.169945 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.169697    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/133b3d42-ffd1-48c0-84ac-252d0201d59f-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:50.169945 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.169710    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/133b3d42-ffd1-48c0-84ac-252d0201d59f-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:50.486890 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.486856    2551 generic.go:356] "Generic (PLEG): container finished" podID="133b3d42-ffd1-48c0-84ac-252d0201d59f" containerID="8e2af03e7e7f95aa110e9e30da92e85a779d6d5547a03209066c0deda0927469" exitCode=0
Aug 01 00:11:50.487084 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.486936    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"
Aug 01 00:11:50.487084 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.486942    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8" event={"ID":"133b3d42-ffd1-48c0-84ac-252d0201d59f","Type":"ContainerDied","Data":"8e2af03e7e7f95aa110e9e30da92e85a779d6d5547a03209066c0deda0927469"}
Aug 01 00:11:50.487084 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.486998    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8" event={"ID":"133b3d42-ffd1-48c0-84ac-252d0201d59f","Type":"ContainerDied","Data":"ab2af80178c4ed321e3b1394ca3a2f7ab027136d7da2ebf2d5d2a242e586b18b"}
Aug 01 00:11:50.487084 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.487022    2551 scope.go:117] "RemoveContainer" containerID="8e2af03e7e7f95aa110e9e30da92e85a779d6d5547a03209066c0deda0927469"
Aug 01 00:11:50.496885 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.496854    2551 scope.go:117] "RemoveContainer" containerID="8e2af03e7e7f95aa110e9e30da92e85a779d6d5547a03209066c0deda0927469"
Aug 01 00:11:50.497548 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:11:50.497520    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2af03e7e7f95aa110e9e30da92e85a779d6d5547a03209066c0deda0927469\": container with ID starting with 8e2af03e7e7f95aa110e9e30da92e85a779d6d5547a03209066c0deda0927469 not found: ID does not exist" containerID="8e2af03e7e7f95aa110e9e30da92e85a779d6d5547a03209066c0deda0927469"
Aug 01 00:11:50.497668 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.497559    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2af03e7e7f95aa110e9e30da92e85a779d6d5547a03209066c0deda0927469"} err="failed to get container status \"8e2af03e7e7f95aa110e9e30da92e85a779d6d5547a03209066c0deda0927469\": rpc error: code = NotFound desc = could not find container \"8e2af03e7e7f95aa110e9e30da92e85a779d6d5547a03209066c0deda0927469\": container with ID starting with 8e2af03e7e7f95aa110e9e30da92e85a779d6d5547a03209066c0deda0927469 not found: ID does not exist"
Aug 01 00:11:50.510369 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.510347    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"]
Aug 01 00:11:50.517739 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:50.517718    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-r5wq8"]
Aug 01 00:11:51.946264 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:51.946236    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="133b3d42-ffd1-48c0-84ac-252d0201d59f" path="/var/lib/kubelet/pods/133b3d42-ffd1-48c0-84ac-252d0201d59f/volumes"
Aug 01 00:11:52.775673 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:52.775641    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:54.049425 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.049382    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr" podUID="8d8999de-60b4-4c8d-98ee-6b2863333c84" containerName="main" containerID="cri-o://a68451dcc518b26c23a6df1a8020ab21660d698bdf80a68104edfc7fb1611a1f" gracePeriod=45
Aug 01 00:11:54.054251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.054216    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr" podUID="8d8999de-60b4-4c8d-98ee-6b2863333c84" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": read tcp 10.133.0.2:45372->10.133.0.43:8000: read: connection reset by peer"
Aug 01 00:11:54.294945 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.294907    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:11:54.296883 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.296867    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-tmp-dir\") pod \"8d8999de-60b4-4c8d-98ee-6b2863333c84\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") "
Aug 01 00:11:54.296965 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.296895    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-dshm\") pod \"8d8999de-60b4-4c8d-98ee-6b2863333c84\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") "
Aug 01 00:11:54.296965 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.296930    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nwhc\" (UniqueName: \"kubernetes.io/projected/8d8999de-60b4-4c8d-98ee-6b2863333c84-kube-api-access-7nwhc\") pod \"8d8999de-60b4-4c8d-98ee-6b2863333c84\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") "
Aug 01 00:11:54.296965 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.296960    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-model-cache\") pod \"8d8999de-60b4-4c8d-98ee-6b2863333c84\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") "
Aug 01 00:11:54.297138 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.297006    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d8999de-60b4-4c8d-98ee-6b2863333c84-tls-certs\") pod \"8d8999de-60b4-4c8d-98ee-6b2863333c84\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") "
Aug 01 00:11:54.297138 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.297044    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-home\") pod \"8d8999de-60b4-4c8d-98ee-6b2863333c84\" (UID: \"8d8999de-60b4-4c8d-98ee-6b2863333c84\") "
Aug 01 00:11:54.297212 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.297143    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d8999de-60b4-4c8d-98ee-6b2863333c84" (UID: "8d8999de-60b4-4c8d-98ee-6b2863333c84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:54.297349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.297327    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-home" (OuterVolumeSpecName: "home") pod "8d8999de-60b4-4c8d-98ee-6b2863333c84" (UID: "8d8999de-60b4-4c8d-98ee-6b2863333c84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:54.297480 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.297360    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d8999de-60b4-4c8d-98ee-6b2863333c84" (UID: "8d8999de-60b4-4c8d-98ee-6b2863333c84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:54.297480 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.297396    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:54.297480 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.297413    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:54.299115 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.299091    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8999de-60b4-4c8d-98ee-6b2863333c84-kube-api-access-7nwhc" (OuterVolumeSpecName: "kube-api-access-7nwhc") pod "8d8999de-60b4-4c8d-98ee-6b2863333c84" (UID: "8d8999de-60b4-4c8d-98ee-6b2863333c84"). InnerVolumeSpecName "kube-api-access-7nwhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:11:54.299203 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.299123    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d8999de-60b4-4c8d-98ee-6b2863333c84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d8999de-60b4-4c8d-98ee-6b2863333c84" (UID: "8d8999de-60b4-4c8d-98ee-6b2863333c84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:11:54.299285 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.299271    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-dshm" (OuterVolumeSpecName: "dshm") pod "8d8999de-60b4-4c8d-98ee-6b2863333c84" (UID: "8d8999de-60b4-4c8d-98ee-6b2863333c84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:11:54.398207 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.398173    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:54.398207 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.398204    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nwhc\" (UniqueName: \"kubernetes.io/projected/8d8999de-60b4-4c8d-98ee-6b2863333c84-kube-api-access-7nwhc\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:54.398407 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.398219    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8999de-60b4-4c8d-98ee-6b2863333c84-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:54.398407 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.398230    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d8999de-60b4-4c8d-98ee-6b2863333c84-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:11:54.504815 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.504781    2551 generic.go:356] "Generic (PLEG): container finished" podID="8d8999de-60b4-4c8d-98ee-6b2863333c84" containerID="a68451dcc518b26c23a6df1a8020ab21660d698bdf80a68104edfc7fb1611a1f" exitCode=0
Aug 01 00:11:54.505052 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.504819    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"
Aug 01 00:11:54.505052 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.504869    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr" event={"ID":"8d8999de-60b4-4c8d-98ee-6b2863333c84","Type":"ContainerDied","Data":"a68451dcc518b26c23a6df1a8020ab21660d698bdf80a68104edfc7fb1611a1f"}
Aug 01 00:11:54.505052 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.504940    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr" event={"ID":"8d8999de-60b4-4c8d-98ee-6b2863333c84","Type":"ContainerDied","Data":"5a5d6c87e4e25e32fe2eb3b1e800d4bd04e75f755c831ea9bf4f92aa2dec678a"}
Aug 01 00:11:54.505052 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.504975    2551 scope.go:117] "RemoveContainer" containerID="a68451dcc518b26c23a6df1a8020ab21660d698bdf80a68104edfc7fb1611a1f"
Aug 01 00:11:54.513566 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.513529    2551 scope.go:117] "RemoveContainer" containerID="a68451dcc518b26c23a6df1a8020ab21660d698bdf80a68104edfc7fb1611a1f"
Aug 01 00:11:54.513795 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:11:54.513775    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68451dcc518b26c23a6df1a8020ab21660d698bdf80a68104edfc7fb1611a1f\": container with ID starting with a68451dcc518b26c23a6df1a8020ab21660d698bdf80a68104edfc7fb1611a1f not found: ID does not exist" containerID="a68451dcc518b26c23a6df1a8020ab21660d698bdf80a68104edfc7fb1611a1f"
Aug 01 00:11:54.513890 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.513806    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68451dcc518b26c23a6df1a8020ab21660d698bdf80a68104edfc7fb1611a1f"} err="failed to get container status \"a68451dcc518b26c23a6df1a8020ab21660d698bdf80a68104edfc7fb1611a1f\": rpc error: code = NotFound desc = could not find container \"a68451dcc518b26c23a6df1a8020ab21660d698bdf80a68104edfc7fb1611a1f\": container with ID starting with a68451dcc518b26c23a6df1a8020ab21660d698bdf80a68104edfc7fb1611a1f not found: ID does not exist"
Aug 01 00:11:54.525216 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.525188    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"]
Aug 01 00:11:54.529391 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:54.529373    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7p6dr"]
Aug 01 00:11:55.946326 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:55.946292    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d8999de-60b4-4c8d-98ee-6b2863333c84" path="/var/lib/kubelet/pods/8d8999de-60b4-4c8d-98ee-6b2863333c84/volumes"
Aug 01 00:11:58.433878 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:58.433843    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:58.550715 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:58.550681    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:58.612154 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:11:58.612121    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:03.998010 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:03.997971    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:04.146818 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:04.146787    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:25.775586 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.775553    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"]
Aug 01 00:12:25.776003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.775852    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d8999de-60b4-4c8d-98ee-6b2863333c84" containerName="main"
Aug 01 00:12:25.776003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.775863    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d8999de-60b4-4c8d-98ee-6b2863333c84" containerName="main"
Aug 01 00:12:25.776003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.775886    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83746e3e-0540-44e5-adf5-878e7c4bc460" containerName="storage-initializer"
Aug 01 00:12:25.776003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.775892    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="83746e3e-0540-44e5-adf5-878e7c4bc460" containerName="storage-initializer"
Aug 01 00:12:25.776003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.775899    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09fa8554-8716-45d1-8028-aeddeb6a3da8" containerName="main"
Aug 01 00:12:25.776003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.775905    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="09fa8554-8716-45d1-8028-aeddeb6a3da8" containerName="main"
Aug 01 00:12:25.776003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.775912    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133b3d42-ffd1-48c0-84ac-252d0201d59f" containerName="main"
Aug 01 00:12:25.776003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.775933    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="133b3d42-ffd1-48c0-84ac-252d0201d59f" containerName="main"
Aug 01 00:12:25.776003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.775941    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83746e3e-0540-44e5-adf5-878e7c4bc460" containerName="main"
Aug 01 00:12:25.776003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.775946    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="83746e3e-0540-44e5-adf5-878e7c4bc460" containerName="main"
Aug 01 00:12:25.776003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.775995    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="09fa8554-8716-45d1-8028-aeddeb6a3da8" containerName="main"
Aug 01 00:12:25.776003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.776004    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d8999de-60b4-4c8d-98ee-6b2863333c84" containerName="main"
Aug 01 00:12:25.776372 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.776013    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="133b3d42-ffd1-48c0-84ac-252d0201d59f" containerName="main"
Aug 01 00:12:25.776372 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.776019    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="83746e3e-0540-44e5-adf5-878e7c4bc460" containerName="main"
Aug 01 00:12:25.779146 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.779123    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.781212 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.781191    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Aug 01 00:12:25.781212 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.781199    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Aug 01 00:12:25.781645 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.781628    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Aug 01 00:12:25.781726 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.781631    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-7mfhv\""
Aug 01 00:12:25.781726 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.781631    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Aug 01 00:12:25.781726 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.781657    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:12:25.790041 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.790022    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"]
Aug 01 00:12:25.805156 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.805131    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:25.861970 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.861943    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-dshm\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.862086 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.862000    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-home\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.862086 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.862023    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/110cbc4d-4441-4cde-afe6-aef6268d4ba7-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.862086 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.862041    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.862086 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.862068    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-model-cache\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.862248 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.862092    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110cbc4d-4441-4cde-afe6-aef6268d4ba7-tls-certs\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.862248 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.862126    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8flx7\" (UniqueName: \"kubernetes.io/projected/110cbc4d-4441-4cde-afe6-aef6268d4ba7-kube-api-access-8flx7\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.862248 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.862176    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-tmp-dir\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.963596 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.963560    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flx7\" (UniqueName: \"kubernetes.io/projected/110cbc4d-4441-4cde-afe6-aef6268d4ba7-kube-api-access-8flx7\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.963775 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.963699    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-tmp-dir\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.963775 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.963747    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-dshm\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.963897 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.963801    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-home\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.963897 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.963831    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/110cbc4d-4441-4cde-afe6-aef6268d4ba7-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.963897 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.963858    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.963897 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.963887    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-model-cache\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.964143 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.963911    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110cbc4d-4441-4cde-afe6-aef6268d4ba7-tls-certs\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.964215 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.964190    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-tmp-dir\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.964356 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.964320    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-home\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.964485 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.964445    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-model-cache\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.964615 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.964593    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.964689 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.964622    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/110cbc4d-4441-4cde-afe6-aef6268d4ba7-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.966216 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.966194    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-dshm\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.966456 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.966439    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110cbc4d-4441-4cde-afe6-aef6268d4ba7-tls-certs\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:25.971051 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:25.971032    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flx7\" (UniqueName: \"kubernetes.io/projected/110cbc4d-4441-4cde-afe6-aef6268d4ba7-kube-api-access-8flx7\") pod \"tokenizer-template-args-test-kserve-57f54c4c47-ddfpm\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:26.062181 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.062106    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"]
Aug 01 00:12:26.066744 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.066707    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"
Aug 01 00:12:26.067505 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.067473    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:26.068874 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.068850    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-r7rfg\""
Aug 01 00:12:26.074112 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.074054    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"]
Aug 01 00:12:26.090250 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.090226    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:26.161806 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.161769    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:26.165485 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.165353    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88m6g\" (UniqueName: \"kubernetes.io/projected/bdca0418-8459-49f0-b3b9-4f452395db96-kube-api-access-88m6g\") pod \"tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp\" (UID: \"bdca0418-8459-49f0-b3b9-4f452395db96\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"
Aug 01 00:12:26.165624 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.165512    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdca0418-8459-49f0-b3b9-4f452395db96-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp\" (UID: \"bdca0418-8459-49f0-b3b9-4f452395db96\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"
Aug 01 00:12:26.222394 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.222362    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"]
Aug 01 00:12:26.225940 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:12:26.225894    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110cbc4d_4441_4cde_afe6_aef6268d4ba7.slice/crio-a68320d0aaa3a5db102f595dfc85397b62fb41d89dcc328159f1c3f790b7801e WatchSource:0}: Error finding container a68320d0aaa3a5db102f595dfc85397b62fb41d89dcc328159f1c3f790b7801e: Status 404 returned error can't find the container with id a68320d0aaa3a5db102f595dfc85397b62fb41d89dcc328159f1c3f790b7801e
Aug 01 00:12:26.266258 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.266231    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88m6g\" (UniqueName: \"kubernetes.io/projected/bdca0418-8459-49f0-b3b9-4f452395db96-kube-api-access-88m6g\") pod \"tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp\" (UID: \"bdca0418-8459-49f0-b3b9-4f452395db96\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"
Aug 01 00:12:26.266352 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.266337    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdca0418-8459-49f0-b3b9-4f452395db96-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp\" (UID: \"bdca0418-8459-49f0-b3b9-4f452395db96\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"
Aug 01 00:12:26.268701 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.268672    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdca0418-8459-49f0-b3b9-4f452395db96-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp\" (UID: \"bdca0418-8459-49f0-b3b9-4f452395db96\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"
Aug 01 00:12:26.273090 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.273069    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88m6g\" (UniqueName: \"kubernetes.io/projected/bdca0418-8459-49f0-b3b9-4f452395db96-kube-api-access-88m6g\") pod \"tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp\" (UID: \"bdca0418-8459-49f0-b3b9-4f452395db96\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"
Aug 01 00:12:26.379635 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.379555    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"
Aug 01 00:12:26.511500 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.511471    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"]
Aug 01 00:12:26.513724 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:12:26.513699    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdca0418_8459_49f0_b3b9_4f452395db96.slice/crio-4477d52439fa25017410c98b6cf4c96acae0dac1e1c5959e3292a26d91c3323d WatchSource:0}: Error finding container 4477d52439fa25017410c98b6cf4c96acae0dac1e1c5959e3292a26d91c3323d: Status 404 returned error can't find the container with id 4477d52439fa25017410c98b6cf4c96acae0dac1e1c5959e3292a26d91c3323d
Aug 01 00:12:26.635774 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.635644    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm" event={"ID":"110cbc4d-4441-4cde-afe6-aef6268d4ba7","Type":"ContainerStarted","Data":"1939f4f093af1e80978d1d3f529f698409cc49b46428038d4d86b558abf8f2ba"}
Aug 01 00:12:26.635774 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.635745    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm" event={"ID":"110cbc4d-4441-4cde-afe6-aef6268d4ba7","Type":"ContainerStarted","Data":"a68320d0aaa3a5db102f595dfc85397b62fb41d89dcc328159f1c3f790b7801e"}
Aug 01 00:12:26.637637 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.637607    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp" event={"ID":"bdca0418-8459-49f0-b3b9-4f452395db96","Type":"ContainerStarted","Data":"3220d4f0dbf479c9ca538d1034f2507b5ee614a0116fdb2d30e1c1b909ccb237"}
Aug 01 00:12:26.637767 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.637660    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp" event={"ID":"bdca0418-8459-49f0-b3b9-4f452395db96","Type":"ContainerStarted","Data":"4477d52439fa25017410c98b6cf4c96acae0dac1e1c5959e3292a26d91c3323d"}
Aug 01 00:12:26.637767 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.637693    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"
Aug 01 00:12:26.668908 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:26.668835    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp" podStartSLOduration=0.668813591 podStartE2EDuration="668.813591ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:12:26.667401661 +0000 UTC m=+1589.361449078" watchObservedRunningTime="2026-08-01 00:12:26.668813591 +0000 UTC m=+1589.362861008"
Aug 01 00:12:28.647457 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:28.647420    2551 generic.go:356] "Generic (PLEG): container finished" podID="110cbc4d-4441-4cde-afe6-aef6268d4ba7" containerID="1939f4f093af1e80978d1d3f529f698409cc49b46428038d4d86b558abf8f2ba" exitCode=0
Aug 01 00:12:28.647834 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:28.647462    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm" event={"ID":"110cbc4d-4441-4cde-afe6-aef6268d4ba7","Type":"ContainerDied","Data":"1939f4f093af1e80978d1d3f529f698409cc49b46428038d4d86b558abf8f2ba"}
Aug 01 00:12:29.633417 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:29.633382    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:29.653425 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:29.653392    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm" event={"ID":"110cbc4d-4441-4cde-afe6-aef6268d4ba7","Type":"ContainerStarted","Data":"ee8a9901fc8817724383a0cc0e4b63849ccb05b16016d817640c9458cade0a22"}
Aug 01 00:12:29.676829 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:29.676784    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm" podStartSLOduration=2.51845511 podStartE2EDuration="4.676771179s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.158316069s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:12:29.675546031 +0000 UTC m=+1592.369593450" watchObservedRunningTime="2026-08-01 00:12:29.676771179 +0000 UTC m=+1592.370818594"
Aug 01 00:12:29.734076 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:29.734048    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:29.837065 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:29.837030    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:30.091078 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:30.091030    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:34.711131 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:34.711083    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:36.091345 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:36.091309    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:36.103582 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:36.103550    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:36.691271 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:36.691240    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:12:42.066743 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:42.066680    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:43.868119 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:43.868087    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:44.454768 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:44.454735    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:57.155216 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.155127    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"]
Aug 01 00:12:57.158617 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.158596    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.161341 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.161316    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-xdzmn\""
Aug 01 00:12:57.161472 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.161342    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Aug 01 00:12:57.161472 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.161357    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Aug 01 00:12:57.161472 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.161405    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Aug 01 00:12:57.168489 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.168469    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"]
Aug 01 00:12:57.188081 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.188055    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:57.227435 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.227407    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.227544 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.227445    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scf79\" (UniqueName: \"kubernetes.io/projected/48210d38-5527-45c1-bf87-f9cf6a14b65a-kube-api-access-scf79\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.227544 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.227480    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-dshm\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.227618 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.227540    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-model-cache\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.227618 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.227580    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-home\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.227618 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.227595    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48210d38-5527-45c1-bf87-f9cf6a14b65a-tls-certs\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.239297 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.239272    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:57.328229 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.328202    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.328391 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.328240    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scf79\" (UniqueName: \"kubernetes.io/projected/48210d38-5527-45c1-bf87-f9cf6a14b65a-kube-api-access-scf79\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.328391 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.328262    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-dshm\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.328391 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.328292    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-model-cache\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.328391 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.328315    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-home\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.328391 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.328329    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48210d38-5527-45c1-bf87-f9cf6a14b65a-tls-certs\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.328663 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.328643    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.328732 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.328699    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-model-cache\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.328822 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.328804    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-home\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.330608 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.330590    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-dshm\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.330826 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.330810    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48210d38-5527-45c1-bf87-f9cf6a14b65a-tls-certs\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.335177 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.335157    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scf79\" (UniqueName: \"kubernetes.io/projected/48210d38-5527-45c1-bf87-f9cf6a14b65a-kube-api-access-scf79\") pod \"test-div-v1-kserve-fb566d667-x2rqx\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.470743 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.470709    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:12:57.607704 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.607676    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"]
Aug 01 00:12:57.611082 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:12:57.611053    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48210d38_5527_45c1_bf87_f9cf6a14b65a.slice/crio-0fe9a3596998615691b83ff8e2340c951fa239ab927e157686d0d183f4b9d18d WatchSource:0}: Error finding container 0fe9a3596998615691b83ff8e2340c951fa239ab927e157686d0d183f4b9d18d: Status 404 returned error can't find the container with id 0fe9a3596998615691b83ff8e2340c951fa239ab927e157686d0d183f4b9d18d
Aug 01 00:12:57.643877 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.643853    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"
Aug 01 00:12:57.771071 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.770989    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx" event={"ID":"48210d38-5527-45c1-bf87-f9cf6a14b65a","Type":"ContainerStarted","Data":"b699882987e38973dcdbf5b1331e0278bc2073d0d7d36dc3b58930944a6df2ac"}
Aug 01 00:12:57.771202 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:57.771078    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx" event={"ID":"48210d38-5527-45c1-bf87-f9cf6a14b65a","Type":"ContainerStarted","Data":"0fe9a3596998615691b83ff8e2340c951fa239ab927e157686d0d183f4b9d18d"}
Aug 01 00:12:58.471534 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:12:58.471501    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:13:00.273162 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:00.273130    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:03.447757 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:03.447719    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:07.471592 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:07.471551    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:13:07.484117 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:07.484090    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:13:07.501797 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:07.501748    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx" podStartSLOduration=10.50173651 podStartE2EDuration="10.50173651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:12:57.789399451 +0000 UTC m=+1620.483446868" watchObservedRunningTime="2026-08-01 00:13:07.50173651 +0000 UTC m=+1630.195783925"
Aug 01 00:13:07.829499 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:07.829420    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:13:11.468274 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:11.468244    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:16.996455 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:16.996417    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:17.098168 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:17.098132    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:24.397808 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:24.397774    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"]
Aug 01 00:13:24.406410 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:24.406381    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:24.406659 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:24.406636    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"]
Aug 01 00:13:24.416522 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:13:24.416483    2551 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:13:24.416522 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:13:24.416522 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:13:24.416522 ip-10-0-131-136 kubenswrapper[2551]:  > containerID="ee8a9901fc8817724383a0cc0e4b63849ccb05b16016d817640c9458cade0a22" cmd=["/bin/sleep","15"]
Aug 01 00:13:24.416845 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:24.416542    2551 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm" message="" err=<
Aug 01 00:13:24.416845 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:13:24.416845 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:13:24.416845 ip-10-0-131-136 kubenswrapper[2551]:  >
Aug 01 00:13:24.416845 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:13:24.416560    2551 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:13:24.416845 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:13:24.416845 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:13:24.416845 ip-10-0-131-136 kubenswrapper[2551]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm" podUID="110cbc4d-4441-4cde-afe6-aef6268d4ba7" containerName="main" containerID="cri-o://ee8a9901fc8817724383a0cc0e4b63849ccb05b16016d817640c9458cade0a22"
Aug 01 00:13:24.416845 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:24.416605    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm" podUID="110cbc4d-4441-4cde-afe6-aef6268d4ba7" containerName="main" containerID="cri-o://ee8a9901fc8817724383a0cc0e4b63849ccb05b16016d817640c9458cade0a22" gracePeriod=60
Aug 01 00:13:24.464833 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:24.464803    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:24.681488 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:24.681398    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm" podUID="110cbc4d-4441-4cde-afe6-aef6268d4ba7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Aug 01 00:13:24.903173 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:24.903141    2551 generic.go:356] "Generic (PLEG): container finished" podID="110cbc4d-4441-4cde-afe6-aef6268d4ba7" containerID="ee8a9901fc8817724383a0cc0e4b63849ccb05b16016d817640c9458cade0a22" exitCode=0
Aug 01 00:13:24.903341 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:24.903211    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm" event={"ID":"110cbc4d-4441-4cde-afe6-aef6268d4ba7","Type":"ContainerDied","Data":"ee8a9901fc8817724383a0cc0e4b63849ccb05b16016d817640c9458cade0a22"}
Aug 01 00:13:25.161269 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.161235    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:13:25.266813 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.266769    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8flx7\" (UniqueName: \"kubernetes.io/projected/110cbc4d-4441-4cde-afe6-aef6268d4ba7-kube-api-access-8flx7\") pod \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") "
Aug 01 00:13:25.267012 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.266819    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-tmp-dir\") pod \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") "
Aug 01 00:13:25.267012 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.266926    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-dshm\") pod \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") "
Aug 01 00:13:25.267139 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.267025    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110cbc4d-4441-4cde-afe6-aef6268d4ba7-tls-certs\") pod \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") "
Aug 01 00:13:25.267139 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.267056    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "110cbc4d-4441-4cde-afe6-aef6268d4ba7" (UID: "110cbc4d-4441-4cde-afe6-aef6268d4ba7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:25.267139 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.267091    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-home\") pod \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") "
Aug 01 00:13:25.267139 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.267118    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-kserve-provision-location\") pod \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") "
Aug 01 00:13:25.267345 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.267161    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-model-cache\") pod \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") "
Aug 01 00:13:25.267345 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.267208    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/110cbc4d-4441-4cde-afe6-aef6268d4ba7-cabundle-cert\") pod \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\" (UID: \"110cbc4d-4441-4cde-afe6-aef6268d4ba7\") "
Aug 01 00:13:25.267950 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.267557    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.267950 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.267576    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-home" (OuterVolumeSpecName: "home") pod "110cbc4d-4441-4cde-afe6-aef6268d4ba7" (UID: "110cbc4d-4441-4cde-afe6-aef6268d4ba7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:25.267950 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.267879    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110cbc4d-4441-4cde-afe6-aef6268d4ba7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "110cbc4d-4441-4cde-afe6-aef6268d4ba7" (UID: "110cbc4d-4441-4cde-afe6-aef6268d4ba7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:13:25.268190 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.267970    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-model-cache" (OuterVolumeSpecName: "model-cache") pod "110cbc4d-4441-4cde-afe6-aef6268d4ba7" (UID: "110cbc4d-4441-4cde-afe6-aef6268d4ba7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:25.278409 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.278388    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110cbc4d-4441-4cde-afe6-aef6268d4ba7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "110cbc4d-4441-4cde-afe6-aef6268d4ba7" (UID: "110cbc4d-4441-4cde-afe6-aef6268d4ba7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:13:25.278620 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.278587    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110cbc4d-4441-4cde-afe6-aef6268d4ba7-kube-api-access-8flx7" (OuterVolumeSpecName: "kube-api-access-8flx7") pod "110cbc4d-4441-4cde-afe6-aef6268d4ba7" (UID: "110cbc4d-4441-4cde-afe6-aef6268d4ba7"). InnerVolumeSpecName "kube-api-access-8flx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:13:25.278700 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.278612    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-dshm" (OuterVolumeSpecName: "dshm") pod "110cbc4d-4441-4cde-afe6-aef6268d4ba7" (UID: "110cbc4d-4441-4cde-afe6-aef6268d4ba7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:25.301876 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.301783    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "110cbc4d-4441-4cde-afe6-aef6268d4ba7" (UID: "110cbc4d-4441-4cde-afe6-aef6268d4ba7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:25.369094 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.369059    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.369094 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.369090    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/110cbc4d-4441-4cde-afe6-aef6268d4ba7-cabundle-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.369291 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.369105    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8flx7\" (UniqueName: \"kubernetes.io/projected/110cbc4d-4441-4cde-afe6-aef6268d4ba7-kube-api-access-8flx7\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.369291 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.369118    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.369291 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.369130    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110cbc4d-4441-4cde-afe6-aef6268d4ba7-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.369291 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.369141    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.369291 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.369150    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/110cbc4d-4441-4cde-afe6-aef6268d4ba7-kserve-provision-location\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:25.909600 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.909566    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"
Aug 01 00:13:25.909600 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.909575    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm" event={"ID":"110cbc4d-4441-4cde-afe6-aef6268d4ba7","Type":"ContainerDied","Data":"a68320d0aaa3a5db102f595dfc85397b62fb41d89dcc328159f1c3f790b7801e"}
Aug 01 00:13:25.910142 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.909687    2551 scope.go:117] "RemoveContainer" containerID="ee8a9901fc8817724383a0cc0e4b63849ccb05b16016d817640c9458cade0a22"
Aug 01 00:13:25.918717 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.918700    2551 scope.go:117] "RemoveContainer" containerID="1939f4f093af1e80978d1d3f529f698409cc49b46428038d4d86b558abf8f2ba"
Aug 01 00:13:25.929296 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.929270    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"]
Aug 01 00:13:25.932814 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.932793    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57f54c4c47-ddfpm"]
Aug 01 00:13:25.946153 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:25.946133    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="110cbc4d-4441-4cde-afe6-aef6268d4ba7" path="/var/lib/kubelet/pods/110cbc4d-4441-4cde-afe6-aef6268d4ba7/volumes"
Aug 01 00:13:27.643772 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:27.643730    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp" podUID="bdca0418-8459-49f0-b3b9-4f452395db96" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:13:29.734701 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:29.734666    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:29.838348 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:29.838300    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:29.942911 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:29.942868    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:29.963945 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:29.963900    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"]
Aug 01 00:13:30.035173 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:30.035077    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:30.196520 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:30.196487    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:30.298741 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:30.298670    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:30.398098 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:30.398067    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:34.812189 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:34.812154    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:35.803056 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:35.803023    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:37.643988 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:37.643949    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp" podUID="bdca0418-8459-49f0-b3b9-4f452395db96" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:13:39.419052 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.419010    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp" podUID="bdca0418-8459-49f0-b3b9-4f452395db96" containerName="main" containerID="cri-o://3220d4f0dbf479c9ca538d1034f2507b5ee614a0116fdb2d30e1c1b909ccb237" gracePeriod=45
Aug 01 00:13:39.665751 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.665722    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"
Aug 01 00:13:39.800171 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.800132    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88m6g\" (UniqueName: \"kubernetes.io/projected/bdca0418-8459-49f0-b3b9-4f452395db96-kube-api-access-88m6g\") pod \"bdca0418-8459-49f0-b3b9-4f452395db96\" (UID: \"bdca0418-8459-49f0-b3b9-4f452395db96\") "
Aug 01 00:13:39.800171 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.800184    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdca0418-8459-49f0-b3b9-4f452395db96-tls-certs\") pod \"bdca0418-8459-49f0-b3b9-4f452395db96\" (UID: \"bdca0418-8459-49f0-b3b9-4f452395db96\") "
Aug 01 00:13:39.802243 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.802220    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdca0418-8459-49f0-b3b9-4f452395db96-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdca0418-8459-49f0-b3b9-4f452395db96" (UID: "bdca0418-8459-49f0-b3b9-4f452395db96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:13:39.802334 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.802270    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdca0418-8459-49f0-b3b9-4f452395db96-kube-api-access-88m6g" (OuterVolumeSpecName: "kube-api-access-88m6g") pod "bdca0418-8459-49f0-b3b9-4f452395db96" (UID: "bdca0418-8459-49f0-b3b9-4f452395db96"). InnerVolumeSpecName "kube-api-access-88m6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:13:39.901595 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.901567    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88m6g\" (UniqueName: \"kubernetes.io/projected/bdca0418-8459-49f0-b3b9-4f452395db96-kube-api-access-88m6g\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:39.901595 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.901596    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdca0418-8459-49f0-b3b9-4f452395db96-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:39.974676 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.974642    2551 generic.go:356] "Generic (PLEG): container finished" podID="bdca0418-8459-49f0-b3b9-4f452395db96" containerID="3220d4f0dbf479c9ca538d1034f2507b5ee614a0116fdb2d30e1c1b909ccb237" exitCode=0
Aug 01 00:13:39.974784 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.974688    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"
Aug 01 00:13:39.974784 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.974682    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp" event={"ID":"bdca0418-8459-49f0-b3b9-4f452395db96","Type":"ContainerDied","Data":"3220d4f0dbf479c9ca538d1034f2507b5ee614a0116fdb2d30e1c1b909ccb237"}
Aug 01 00:13:39.974870 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.974801    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp" event={"ID":"bdca0418-8459-49f0-b3b9-4f452395db96","Type":"ContainerDied","Data":"4477d52439fa25017410c98b6cf4c96acae0dac1e1c5959e3292a26d91c3323d"}
Aug 01 00:13:39.974870 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.974824    2551 scope.go:117] "RemoveContainer" containerID="3220d4f0dbf479c9ca538d1034f2507b5ee614a0116fdb2d30e1c1b909ccb237"
Aug 01 00:13:39.984703 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.984682    2551 scope.go:117] "RemoveContainer" containerID="3220d4f0dbf479c9ca538d1034f2507b5ee614a0116fdb2d30e1c1b909ccb237"
Aug 01 00:13:39.985023 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:13:39.985000    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3220d4f0dbf479c9ca538d1034f2507b5ee614a0116fdb2d30e1c1b909ccb237\": container with ID starting with 3220d4f0dbf479c9ca538d1034f2507b5ee614a0116fdb2d30e1c1b909ccb237 not found: ID does not exist" containerID="3220d4f0dbf479c9ca538d1034f2507b5ee614a0116fdb2d30e1c1b909ccb237"
Aug 01 00:13:39.985107 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.985038    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3220d4f0dbf479c9ca538d1034f2507b5ee614a0116fdb2d30e1c1b909ccb237"} err="failed to get container status \"3220d4f0dbf479c9ca538d1034f2507b5ee614a0116fdb2d30e1c1b909ccb237\": rpc error: code = NotFound desc = could not find container \"3220d4f0dbf479c9ca538d1034f2507b5ee614a0116fdb2d30e1c1b909ccb237\": container with ID starting with 3220d4f0dbf479c9ca538d1034f2507b5ee614a0116fdb2d30e1c1b909ccb237 not found: ID does not exist"
Aug 01 00:13:39.989248 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.989218    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"]
Aug 01 00:13:39.993098 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:39.993075    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-6b477rplpp"]
Aug 01 00:13:41.259650 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:41.259616    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:41.349939 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:41.349892    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:41.946236 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:41.946208    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdca0418-8459-49f0-b3b9-4f452395db96" path="/var/lib/kubelet/pods/bdca0418-8459-49f0-b3b9-4f452395db96/volumes"
Aug 01 00:13:42.988021 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:42.987985    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:43.191853 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:43.191819    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:43.288913 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:43.288837    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:44.982443 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:44.982405    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx" podUID="48210d38-5527-45c1-bf87-f9cf6a14b65a" containerName="main" containerID="cri-o://b699882987e38973dcdbf5b1331e0278bc2073d0d7d36dc3b58930944a6df2ac" gracePeriod=45
Aug 01 00:13:45.224771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.224745    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:13:45.248229 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.248161    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-model-cache\") pod \"48210d38-5527-45c1-bf87-f9cf6a14b65a\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") "
Aug 01 00:13:45.248449 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.248404    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-model-cache" (OuterVolumeSpecName: "model-cache") pod "48210d38-5527-45c1-bf87-f9cf6a14b65a" (UID: "48210d38-5527-45c1-bf87-f9cf6a14b65a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:45.248449 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.248426    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scf79\" (UniqueName: \"kubernetes.io/projected/48210d38-5527-45c1-bf87-f9cf6a14b65a-kube-api-access-scf79\") pod \"48210d38-5527-45c1-bf87-f9cf6a14b65a\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") "
Aug 01 00:13:45.248644 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.248487    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-home\") pod \"48210d38-5527-45c1-bf87-f9cf6a14b65a\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") "
Aug 01 00:13:45.248644 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.248515    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48210d38-5527-45c1-bf87-f9cf6a14b65a-tls-certs\") pod \"48210d38-5527-45c1-bf87-f9cf6a14b65a\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") "
Aug 01 00:13:45.248644 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.248551    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-tmp-dir\") pod \"48210d38-5527-45c1-bf87-f9cf6a14b65a\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") "
Aug 01 00:13:45.248644 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.248571    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-dshm\") pod \"48210d38-5527-45c1-bf87-f9cf6a14b65a\" (UID: \"48210d38-5527-45c1-bf87-f9cf6a14b65a\") "
Aug 01 00:13:45.248865 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.248812    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:45.249231 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.249051    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-home" (OuterVolumeSpecName: "home") pod "48210d38-5527-45c1-bf87-f9cf6a14b65a" (UID: "48210d38-5527-45c1-bf87-f9cf6a14b65a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:45.249525 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.249393    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48210d38-5527-45c1-bf87-f9cf6a14b65a" (UID: "48210d38-5527-45c1-bf87-f9cf6a14b65a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:45.251212 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.251185    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48210d38-5527-45c1-bf87-f9cf6a14b65a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48210d38-5527-45c1-bf87-f9cf6a14b65a" (UID: "48210d38-5527-45c1-bf87-f9cf6a14b65a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:13:45.251427 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.251401    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-dshm" (OuterVolumeSpecName: "dshm") pod "48210d38-5527-45c1-bf87-f9cf6a14b65a" (UID: "48210d38-5527-45c1-bf87-f9cf6a14b65a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:13:45.251781 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.251760    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48210d38-5527-45c1-bf87-f9cf6a14b65a-kube-api-access-scf79" (OuterVolumeSpecName: "kube-api-access-scf79") pod "48210d38-5527-45c1-bf87-f9cf6a14b65a" (UID: "48210d38-5527-45c1-bf87-f9cf6a14b65a"). InnerVolumeSpecName "kube-api-access-scf79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:13:45.349426 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.349380    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scf79\" (UniqueName: \"kubernetes.io/projected/48210d38-5527-45c1-bf87-f9cf6a14b65a-kube-api-access-scf79\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:45.349426 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.349420    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:45.349756 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.349436    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48210d38-5527-45c1-bf87-f9cf6a14b65a-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:45.349756 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.349479    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:45.349756 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:45.349494    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48210d38-5527-45c1-bf87-f9cf6a14b65a-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:13:46.005272 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:46.005236    2551 generic.go:356] "Generic (PLEG): container finished" podID="48210d38-5527-45c1-bf87-f9cf6a14b65a" containerID="b699882987e38973dcdbf5b1331e0278bc2073d0d7d36dc3b58930944a6df2ac" exitCode=0
Aug 01 00:13:46.005272 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:46.005270    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"
Aug 01 00:13:46.005700 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:46.005296    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx" event={"ID":"48210d38-5527-45c1-bf87-f9cf6a14b65a","Type":"ContainerDied","Data":"b699882987e38973dcdbf5b1331e0278bc2073d0d7d36dc3b58930944a6df2ac"}
Aug 01 00:13:46.005700 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:46.005346    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx" event={"ID":"48210d38-5527-45c1-bf87-f9cf6a14b65a","Type":"ContainerDied","Data":"0fe9a3596998615691b83ff8e2340c951fa239ab927e157686d0d183f4b9d18d"}
Aug 01 00:13:46.005700 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:46.005378    2551 scope.go:117] "RemoveContainer" containerID="b699882987e38973dcdbf5b1331e0278bc2073d0d7d36dc3b58930944a6df2ac"
Aug 01 00:13:46.013443 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:46.013409    2551 scope.go:117] "RemoveContainer" containerID="b699882987e38973dcdbf5b1331e0278bc2073d0d7d36dc3b58930944a6df2ac"
Aug 01 00:13:46.013677 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:13:46.013656    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b699882987e38973dcdbf5b1331e0278bc2073d0d7d36dc3b58930944a6df2ac\": container with ID starting with b699882987e38973dcdbf5b1331e0278bc2073d0d7d36dc3b58930944a6df2ac not found: ID does not exist" containerID="b699882987e38973dcdbf5b1331e0278bc2073d0d7d36dc3b58930944a6df2ac"
Aug 01 00:13:46.013732 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:46.013686    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b699882987e38973dcdbf5b1331e0278bc2073d0d7d36dc3b58930944a6df2ac"} err="failed to get container status \"b699882987e38973dcdbf5b1331e0278bc2073d0d7d36dc3b58930944a6df2ac\": rpc error: code = NotFound desc = could not find container \"b699882987e38973dcdbf5b1331e0278bc2073d0d7d36dc3b58930944a6df2ac\": container with ID starting with b699882987e38973dcdbf5b1331e0278bc2073d0d7d36dc3b58930944a6df2ac not found: ID does not exist"
Aug 01 00:13:46.020864 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:46.020841    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"]
Aug 01 00:13:46.025003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:46.024983    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-x2rqx"]
Aug 01 00:13:47.452994 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.452951    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"]
Aug 01 00:13:47.453454 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.453383    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdca0418-8459-49f0-b3b9-4f452395db96" containerName="main"
Aug 01 00:13:47.453454 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.453398    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdca0418-8459-49f0-b3b9-4f452395db96" containerName="main"
Aug 01 00:13:47.453454 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.453409    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110cbc4d-4441-4cde-afe6-aef6268d4ba7" containerName="main"
Aug 01 00:13:47.453454 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.453417    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="110cbc4d-4441-4cde-afe6-aef6268d4ba7" containerName="main"
Aug 01 00:13:47.453454 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.453437    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110cbc4d-4441-4cde-afe6-aef6268d4ba7" containerName="storage-initializer"
Aug 01 00:13:47.453454 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.453445    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="110cbc4d-4441-4cde-afe6-aef6268d4ba7" containerName="storage-initializer"
Aug 01 00:13:47.453822 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.453461    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48210d38-5527-45c1-bf87-f9cf6a14b65a" containerName="main"
Aug 01 00:13:47.453822 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.453469    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="48210d38-5527-45c1-bf87-f9cf6a14b65a" containerName="main"
Aug 01 00:13:47.453822 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.453534    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdca0418-8459-49f0-b3b9-4f452395db96" containerName="main"
Aug 01 00:13:47.453822 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.453546    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="110cbc4d-4441-4cde-afe6-aef6268d4ba7" containerName="main"
Aug 01 00:13:47.453822 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.453556    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="48210d38-5527-45c1-bf87-f9cf6a14b65a" containerName="main"
Aug 01 00:13:47.462330 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.462306    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.465771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.465055    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Aug 01 00:13:47.465771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.465102    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Aug 01 00:13:47.465771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.465166    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Aug 01 00:13:47.465771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.465116    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-nvh4p\""
Aug 01 00:13:47.466236 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.466214    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"]
Aug 01 00:13:47.484201 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.484176    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:47.565956 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.565905    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-home\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.565956 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.565957    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-dshm\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.566148 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.566028    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.566148 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.566074    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwrds\" (UniqueName: \"kubernetes.io/projected/8a849805-fe3f-4daf-9982-149fc9ed6c6d-kube-api-access-xwrds\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.566148 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.566130    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a849805-fe3f-4daf-9982-149fc9ed6c6d-tls-certs\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.566249 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.566181    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-model-cache\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.666556 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.666524    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-model-cache\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.666742 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.666562    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-home\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.666742 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.666580    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-dshm\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.666742 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.666613    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.666742 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.666647    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrds\" (UniqueName: \"kubernetes.io/projected/8a849805-fe3f-4daf-9982-149fc9ed6c6d-kube-api-access-xwrds\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.666742 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.666688    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a849805-fe3f-4daf-9982-149fc9ed6c6d-tls-certs\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.667043 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.666983    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-model-cache\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.667043 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.667027    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-home\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.667137 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.667114    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.668996 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.668965    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-dshm\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.669119 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.669102    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a849805-fe3f-4daf-9982-149fc9ed6c6d-tls-certs\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.674555 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.674532    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrds\" (UniqueName: \"kubernetes.io/projected/8a849805-fe3f-4daf-9982-149fc9ed6c6d-kube-api-access-xwrds\") pod \"stop-v2-kserve-7f68747c45-cpg85\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.774104 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.774083    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:47.906098 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.906073    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"]
Aug 01 00:13:47.909998 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:13:47.909952    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a849805_fe3f_4daf_9982_149fc9ed6c6d.slice/crio-939d35a4cec81cf51a73bfe579c3b5de9577c45e7775334f984d5b3fcf8c1d5f WatchSource:0}: Error finding container 939d35a4cec81cf51a73bfe579c3b5de9577c45e7775334f984d5b3fcf8c1d5f: Status 404 returned error can't find the container with id 939d35a4cec81cf51a73bfe579c3b5de9577c45e7775334f984d5b3fcf8c1d5f
Aug 01 00:13:47.947838 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:47.947807    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48210d38-5527-45c1-bf87-f9cf6a14b65a" path="/var/lib/kubelet/pods/48210d38-5527-45c1-bf87-f9cf6a14b65a/volumes"
Aug 01 00:13:48.016762 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:48.016670    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85" event={"ID":"8a849805-fe3f-4daf-9982-149fc9ed6c6d","Type":"ContainerStarted","Data":"6ef734db6d513ec7975ca8e949cf4acfed0a564cff10c7e8fa5b5ff0fb04210f"}
Aug 01 00:13:48.016762 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:48.016731    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85" event={"ID":"8a849805-fe3f-4daf-9982-149fc9ed6c6d","Type":"ContainerStarted","Data":"939d35a4cec81cf51a73bfe579c3b5de9577c45e7775334f984d5b3fcf8c1d5f"}
Aug 01 00:13:48.036074 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:48.035980    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85" podStartSLOduration=1.03596393 podStartE2EDuration="1.03596393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:13:48.034117779 +0000 UTC m=+1670.728165196" watchObservedRunningTime="2026-08-01 00:13:48.03596393 +0000 UTC m=+1670.730011346"
Aug 01 00:13:48.774238 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:48.774201    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:49.621033 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:49.620997    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:55.354429 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:55.354394    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:55.454568 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:55.454529    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:57.774484 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:57.774450    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:57.786520 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:57.786495    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:13:58.071188 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:13:58.071120    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:14:11.786318 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:11.786280    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:15.084039 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.084006    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:15.364371 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.364297    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"]
Aug 01 00:14:15.373429 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.372790    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"
Aug 01 00:14:15.374472 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.374448    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"]
Aug 01 00:14:15.376088 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.376065    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Aug 01 00:14:15.376282 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.376257    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-xxjn4\""
Aug 01 00:14:15.376372 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.376149    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Aug 01 00:14:15.376443 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.376162    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-l5g76\""
Aug 01 00:14:15.376508 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.376108    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Aug 01 00:14:15.400576 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.400536    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:15.498332 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.498308    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97300b5c-045d-4a1f-ba68-362a6f9b0828-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw\" (UID: \"97300b5c-045d-4a1f-ba68-362a6f9b0828\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"
Aug 01 00:14:15.498458 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.498348    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9br9\" (UniqueName: \"kubernetes.io/projected/97300b5c-045d-4a1f-ba68-362a6f9b0828-kube-api-access-m9br9\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw\" (UID: \"97300b5c-045d-4a1f-ba68-362a6f9b0828\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"
Aug 01 00:14:15.599771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.599737    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97300b5c-045d-4a1f-ba68-362a6f9b0828-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw\" (UID: \"97300b5c-045d-4a1f-ba68-362a6f9b0828\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"
Aug 01 00:14:15.599961 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.599787    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9br9\" (UniqueName: \"kubernetes.io/projected/97300b5c-045d-4a1f-ba68-362a6f9b0828-kube-api-access-m9br9\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw\" (UID: \"97300b5c-045d-4a1f-ba68-362a6f9b0828\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"
Aug 01 00:14:15.602265 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.602243    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97300b5c-045d-4a1f-ba68-362a6f9b0828-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw\" (UID: \"97300b5c-045d-4a1f-ba68-362a6f9b0828\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"
Aug 01 00:14:15.607329 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.607300    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9br9\" (UniqueName: \"kubernetes.io/projected/97300b5c-045d-4a1f-ba68-362a6f9b0828-kube-api-access-m9br9\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw\" (UID: \"97300b5c-045d-4a1f-ba68-362a6f9b0828\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"
Aug 01 00:14:15.685267 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.685199    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"
Aug 01 00:14:15.814724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:15.814697    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"]
Aug 01 00:14:15.817195 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:14:15.817164    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97300b5c_045d_4a1f_ba68_362a6f9b0828.slice/crio-db0a6fb2eaf0dd76779198c4b8252e8cd0645a56108ba90e221840382a51344c WatchSource:0}: Error finding container db0a6fb2eaf0dd76779198c4b8252e8cd0645a56108ba90e221840382a51344c: Status 404 returned error can't find the container with id db0a6fb2eaf0dd76779198c4b8252e8cd0645a56108ba90e221840382a51344c
Aug 01 00:14:16.136893 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:16.136851    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw" event={"ID":"97300b5c-045d-4a1f-ba68-362a6f9b0828","Type":"ContainerStarted","Data":"1a24966930e059ddb9b7db841cc4e6a10086244e7a128e71563b8d992d2474b5"}
Aug 01 00:14:16.137472 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:16.136934    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw" event={"ID":"97300b5c-045d-4a1f-ba68-362a6f9b0828","Type":"ContainerStarted","Data":"db0a6fb2eaf0dd76779198c4b8252e8cd0645a56108ba90e221840382a51344c"}
Aug 01 00:14:16.137472 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:16.136971    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"
Aug 01 00:14:16.153694 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:16.153636    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw" podStartSLOduration=1.153620779 podStartE2EDuration="1.153620779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:14:16.151462251 +0000 UTC m=+1698.845509667" watchObservedRunningTime="2026-08-01 00:14:16.153620779 +0000 UTC m=+1698.847668194"
Aug 01 00:14:17.344896 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:17.344859    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:17.445685 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:17.445625    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:26.568008 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:26.567899    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:41.331386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:41.331351    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"]
Aug 01 00:14:41.359179 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:41.359150    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:47.143400 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:47.143369    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"
Aug 01 00:14:56.350982 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.350895    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85" podUID="8a849805-fe3f-4daf-9982-149fc9ed6c6d" containerName="main" containerID="cri-o://6ef734db6d513ec7975ca8e949cf4acfed0a564cff10c7e8fa5b5ff0fb04210f" gracePeriod=45
Aug 01 00:14:56.593809 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.593781    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:14:56.661814 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.661729    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-tmp-dir\") pod \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") "
Aug 01 00:14:56.661814 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.661795    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-dshm\") pod \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") "
Aug 01 00:14:56.662066 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.661829    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-home\") pod \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") "
Aug 01 00:14:56.662066 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.661857    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwrds\" (UniqueName: \"kubernetes.io/projected/8a849805-fe3f-4daf-9982-149fc9ed6c6d-kube-api-access-xwrds\") pod \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") "
Aug 01 00:14:56.662066 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.661879    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a849805-fe3f-4daf-9982-149fc9ed6c6d-tls-certs\") pod \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") "
Aug 01 00:14:56.662066 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.661968    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-model-cache\") pod \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\" (UID: \"8a849805-fe3f-4daf-9982-149fc9ed6c6d\") "
Aug 01 00:14:56.662066 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.662016    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a849805-fe3f-4daf-9982-149fc9ed6c6d" (UID: "8a849805-fe3f-4daf-9982-149fc9ed6c6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:14:56.662317 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.662133    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-home" (OuterVolumeSpecName: "home") pod "8a849805-fe3f-4daf-9982-149fc9ed6c6d" (UID: "8a849805-fe3f-4daf-9982-149fc9ed6c6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:14:56.662317 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.662223    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a849805-fe3f-4daf-9982-149fc9ed6c6d" (UID: "8a849805-fe3f-4daf-9982-149fc9ed6c6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:14:56.662317 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.662279    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:14:56.662317 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.662293    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:14:56.664086 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.664062    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a849805-fe3f-4daf-9982-149fc9ed6c6d-kube-api-access-xwrds" (OuterVolumeSpecName: "kube-api-access-xwrds") pod "8a849805-fe3f-4daf-9982-149fc9ed6c6d" (UID: "8a849805-fe3f-4daf-9982-149fc9ed6c6d"). InnerVolumeSpecName "kube-api-access-xwrds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:14:56.664203 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.664077    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a849805-fe3f-4daf-9982-149fc9ed6c6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a849805-fe3f-4daf-9982-149fc9ed6c6d" (UID: "8a849805-fe3f-4daf-9982-149fc9ed6c6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:14:56.664203 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.664088    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-dshm" (OuterVolumeSpecName: "dshm") pod "8a849805-fe3f-4daf-9982-149fc9ed6c6d" (UID: "8a849805-fe3f-4daf-9982-149fc9ed6c6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:14:56.763544 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.763500    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:14:56.763544 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.763538    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a849805-fe3f-4daf-9982-149fc9ed6c6d-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:14:56.763544 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.763548    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwrds\" (UniqueName: \"kubernetes.io/projected/8a849805-fe3f-4daf-9982-149fc9ed6c6d-kube-api-access-xwrds\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:14:56.763544 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:56.763557    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a849805-fe3f-4daf-9982-149fc9ed6c6d-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:14:57.309886 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:57.309850    2551 generic.go:356] "Generic (PLEG): container finished" podID="8a849805-fe3f-4daf-9982-149fc9ed6c6d" containerID="6ef734db6d513ec7975ca8e949cf4acfed0a564cff10c7e8fa5b5ff0fb04210f" exitCode=0
Aug 01 00:14:57.310069 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:57.309892    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"
Aug 01 00:14:57.310069 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:57.309939    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85" event={"ID":"8a849805-fe3f-4daf-9982-149fc9ed6c6d","Type":"ContainerDied","Data":"6ef734db6d513ec7975ca8e949cf4acfed0a564cff10c7e8fa5b5ff0fb04210f"}
Aug 01 00:14:57.310069 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:57.309988    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85" event={"ID":"8a849805-fe3f-4daf-9982-149fc9ed6c6d","Type":"ContainerDied","Data":"939d35a4cec81cf51a73bfe579c3b5de9577c45e7775334f984d5b3fcf8c1d5f"}
Aug 01 00:14:57.310069 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:57.310013    2551 scope.go:117] "RemoveContainer" containerID="6ef734db6d513ec7975ca8e949cf4acfed0a564cff10c7e8fa5b5ff0fb04210f"
Aug 01 00:14:57.318499 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:57.318481    2551 scope.go:117] "RemoveContainer" containerID="6ef734db6d513ec7975ca8e949cf4acfed0a564cff10c7e8fa5b5ff0fb04210f"
Aug 01 00:14:57.318751 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:14:57.318726    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef734db6d513ec7975ca8e949cf4acfed0a564cff10c7e8fa5b5ff0fb04210f\": container with ID starting with 6ef734db6d513ec7975ca8e949cf4acfed0a564cff10c7e8fa5b5ff0fb04210f not found: ID does not exist" containerID="6ef734db6d513ec7975ca8e949cf4acfed0a564cff10c7e8fa5b5ff0fb04210f"
Aug 01 00:14:57.318852 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:57.318755    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef734db6d513ec7975ca8e949cf4acfed0a564cff10c7e8fa5b5ff0fb04210f"} err="failed to get container status \"6ef734db6d513ec7975ca8e949cf4acfed0a564cff10c7e8fa5b5ff0fb04210f\": rpc error: code = NotFound desc = could not find container \"6ef734db6d513ec7975ca8e949cf4acfed0a564cff10c7e8fa5b5ff0fb04210f\": container with ID starting with 6ef734db6d513ec7975ca8e949cf4acfed0a564cff10c7e8fa5b5ff0fb04210f not found: ID does not exist"
Aug 01 00:14:57.329308 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:57.329284    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"]
Aug 01 00:14:57.333992 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:57.333973    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-cpg85"]
Aug 01 00:14:57.946535 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:14:57.946506    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a849805-fe3f-4daf-9982-149fc9ed6c6d" path="/var/lib/kubelet/pods/8a849805-fe3f-4daf-9982-149fc9ed6c6d/volumes"
Aug 01 00:15:03.110212 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:03.110175    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:03.219826 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:03.219762    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:03.277462 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:03.277427    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:08.188931 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:08.188890    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:13.201654 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:13.201611    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"]
Aug 01 00:15:13.203995 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:13.203965    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:13.262882 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:13.262848    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:14.153161 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:14.153120    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:17.144034 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:17.143996    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw" podUID="97300b5c-045d-4a1f-ba68-362a6f9b0828" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:15:25.464766 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:25.464726    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:27.143986 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:27.143944    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw" podUID="97300b5c-045d-4a1f-ba68-362a6f9b0828" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:15:28.222344 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.222312    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw" podUID="97300b5c-045d-4a1f-ba68-362a6f9b0828" containerName="main" containerID="cri-o://1a24966930e059ddb9b7db841cc4e6a10086244e7a128e71563b8d992d2474b5" gracePeriod=45
Aug 01 00:15:28.435096 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.435064    2551 generic.go:356] "Generic (PLEG): container finished" podID="97300b5c-045d-4a1f-ba68-362a6f9b0828" containerID="1a24966930e059ddb9b7db841cc4e6a10086244e7a128e71563b8d992d2474b5" exitCode=0
Aug 01 00:15:28.435250 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.435143    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw" event={"ID":"97300b5c-045d-4a1f-ba68-362a6f9b0828","Type":"ContainerDied","Data":"1a24966930e059ddb9b7db841cc4e6a10086244e7a128e71563b8d992d2474b5"}
Aug 01 00:15:28.467013 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.466986    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"
Aug 01 00:15:28.564146 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.564056    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"]
Aug 01 00:15:28.564445 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.564426    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97300b5c-045d-4a1f-ba68-362a6f9b0828" containerName="main"
Aug 01 00:15:28.564527 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.564447    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="97300b5c-045d-4a1f-ba68-362a6f9b0828" containerName="main"
Aug 01 00:15:28.564527 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.564467    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a849805-fe3f-4daf-9982-149fc9ed6c6d" containerName="main"
Aug 01 00:15:28.564527 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.564475    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a849805-fe3f-4daf-9982-149fc9ed6c6d" containerName="main"
Aug 01 00:15:28.564669 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.564568    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="97300b5c-045d-4a1f-ba68-362a6f9b0828" containerName="main"
Aug 01 00:15:28.564669 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.564583    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a849805-fe3f-4daf-9982-149fc9ed6c6d" containerName="main"
Aug 01 00:15:28.567387 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.567366    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.569396 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.569367    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Aug 01 00:15:28.569502 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.569393    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Aug 01 00:15:28.569502 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.569393    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-mdm6t\""
Aug 01 00:15:28.569685 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.569670    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Aug 01 00:15:28.576174 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.576154    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"]
Aug 01 00:15:28.631801 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.631775    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9br9\" (UniqueName: \"kubernetes.io/projected/97300b5c-045d-4a1f-ba68-362a6f9b0828-kube-api-access-m9br9\") pod \"97300b5c-045d-4a1f-ba68-362a6f9b0828\" (UID: \"97300b5c-045d-4a1f-ba68-362a6f9b0828\") "
Aug 01 00:15:28.631973 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.631808    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97300b5c-045d-4a1f-ba68-362a6f9b0828-tls-certs\") pod \"97300b5c-045d-4a1f-ba68-362a6f9b0828\" (UID: \"97300b5c-045d-4a1f-ba68-362a6f9b0828\") "
Aug 01 00:15:28.634119 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.634090    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97300b5c-045d-4a1f-ba68-362a6f9b0828-kube-api-access-m9br9" (OuterVolumeSpecName: "kube-api-access-m9br9") pod "97300b5c-045d-4a1f-ba68-362a6f9b0828" (UID: "97300b5c-045d-4a1f-ba68-362a6f9b0828"). InnerVolumeSpecName "kube-api-access-m9br9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:15:28.634268 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.634147    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97300b5c-045d-4a1f-ba68-362a6f9b0828-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97300b5c-045d-4a1f-ba68-362a6f9b0828" (UID: "97300b5c-045d-4a1f-ba68-362a6f9b0828"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:15:28.656253 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.656228    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:28.732864 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.732835    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-dshm\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.732864 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.732867    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-tmp-dir\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.733057 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.732891    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcbn\" (UniqueName: \"kubernetes.io/projected/7d026bb6-05c1-41c0-8319-59655b15433d-kube-api-access-bjcbn\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.733057 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.732981    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-home\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.733057 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.733024    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d026bb6-05c1-41c0-8319-59655b15433d-tls-certs\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.733057 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.733054    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-model-cache\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.733278 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.733122    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9br9\" (UniqueName: \"kubernetes.io/projected/97300b5c-045d-4a1f-ba68-362a6f9b0828-kube-api-access-m9br9\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:15:28.733278 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.733141    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97300b5c-045d-4a1f-ba68-362a6f9b0828-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:15:28.833737 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.833650    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d026bb6-05c1-41c0-8319-59655b15433d-tls-certs\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.833737 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.833685    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-model-cache\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.833737 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.833736    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-dshm\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.834042 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.833758    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-tmp-dir\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.834042 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.833784    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcbn\" (UniqueName: \"kubernetes.io/projected/7d026bb6-05c1-41c0-8319-59655b15433d-kube-api-access-bjcbn\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.834042 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.833811    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-home\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.834266 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.834247    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-model-cache\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.834346 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.834299    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-home\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.834346 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.834331    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-tmp-dir\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.836033 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.836013    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-dshm\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.836157 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.836141    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d026bb6-05c1-41c0-8319-59655b15433d-tls-certs\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.840945 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.840907    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcbn\" (UniqueName: \"kubernetes.io/projected/7d026bb6-05c1-41c0-8319-59655b15433d-kube-api-access-bjcbn\") pod \"decom-v2-kserve-77f867d787-xntgn\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:28.878992 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:28.878963    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:29.041605 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:29.041574    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"]
Aug 01 00:15:29.045389 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:15:29.045356    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d026bb6_05c1_41c0_8319_59655b15433d.slice/crio-021e672ab7d581bcc96f7bb9cc76f2c76adb213d47bc7398852fcf5ba97b3b18 WatchSource:0}: Error finding container 021e672ab7d581bcc96f7bb9cc76f2c76adb213d47bc7398852fcf5ba97b3b18: Status 404 returned error can't find the container with id 021e672ab7d581bcc96f7bb9cc76f2c76adb213d47bc7398852fcf5ba97b3b18
Aug 01 00:15:29.445571 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:29.445475    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn" event={"ID":"7d026bb6-05c1-41c0-8319-59655b15433d","Type":"ContainerStarted","Data":"df83970e94a40a880a04537fff99b12511ccb951a79f11d424536c097f9c432f"}
Aug 01 00:15:29.445571 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:29.445565    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn" event={"ID":"7d026bb6-05c1-41c0-8319-59655b15433d","Type":"ContainerStarted","Data":"021e672ab7d581bcc96f7bb9cc76f2c76adb213d47bc7398852fcf5ba97b3b18"}
Aug 01 00:15:29.446869 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:29.446848    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"
Aug 01 00:15:29.446869 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:29.446858    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw" event={"ID":"97300b5c-045d-4a1f-ba68-362a6f9b0828","Type":"ContainerDied","Data":"db0a6fb2eaf0dd76779198c4b8252e8cd0645a56108ba90e221840382a51344c"}
Aug 01 00:15:29.447043 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:29.446905    2551 scope.go:117] "RemoveContainer" containerID="1a24966930e059ddb9b7db841cc4e6a10086244e7a128e71563b8d992d2474b5"
Aug 01 00:15:29.462215 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:29.462170    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn" podStartSLOduration=1.46215543 podStartE2EDuration="1.46215543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:15:29.460720763 +0000 UTC m=+1772.154768180" watchObservedRunningTime="2026-08-01 00:15:29.46215543 +0000 UTC m=+1772.156202897"
Aug 01 00:15:29.470979 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:29.470953    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"]
Aug 01 00:15:29.472321 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:29.472301    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezr4qw"]
Aug 01 00:15:29.879476 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:29.879449    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:29.947634 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:29.947597    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97300b5c-045d-4a1f-ba68-362a6f9b0828" path="/var/lib/kubelet/pods/97300b5c-045d-4a1f-ba68-362a6f9b0828/volumes"
Aug 01 00:15:32.652730 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:32.652697    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:32.751608 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:32.751578    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:38.523813 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:38.523782    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:38.880174 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:38.880084    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:38.892567 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:38.892535    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:39.500016 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:39.499987    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:15:44.035192 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:44.035154    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:44.092834 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:44.092802    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:45.161786 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:45.161747    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:50.769336 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:50.769245    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:50.823762 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:50.823730    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:59.290763 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.290727    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:59.464679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.464642    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"]
Aug 01 00:15:59.469365 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.469337    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"
Aug 01 00:15:59.472369 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.472333    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Aug 01 00:15:59.472599 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.472579    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Aug 01 00:15:59.472716 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.472649    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Aug 01 00:15:59.472716 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.472706    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-t5l4c\""
Aug 01 00:15:59.472838 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.472719    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-4b947\""
Aug 01 00:15:59.475478 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.475457    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"]
Aug 01 00:15:59.494456 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.494426    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:59.587726 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.587640    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5dde6d6-d08d-450b-8fcc-db093c09c8a0-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv\" (UID: \"e5dde6d6-d08d-450b-8fcc-db093c09c8a0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"
Aug 01 00:15:59.587726 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.587693    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw64l\" (UniqueName: \"kubernetes.io/projected/e5dde6d6-d08d-450b-8fcc-db093c09c8a0-kube-api-access-tw64l\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv\" (UID: \"e5dde6d6-d08d-450b-8fcc-db093c09c8a0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"
Aug 01 00:15:59.688217 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.688181    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw64l\" (UniqueName: \"kubernetes.io/projected/e5dde6d6-d08d-450b-8fcc-db093c09c8a0-kube-api-access-tw64l\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv\" (UID: \"e5dde6d6-d08d-450b-8fcc-db093c09c8a0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"
Aug 01 00:15:59.688397 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.688268    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5dde6d6-d08d-450b-8fcc-db093c09c8a0-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv\" (UID: \"e5dde6d6-d08d-450b-8fcc-db093c09c8a0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"
Aug 01 00:15:59.690887 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.690853    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5dde6d6-d08d-450b-8fcc-db093c09c8a0-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv\" (UID: \"e5dde6d6-d08d-450b-8fcc-db093c09c8a0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"
Aug 01 00:15:59.695385 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.695359    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw64l\" (UniqueName: \"kubernetes.io/projected/e5dde6d6-d08d-450b-8fcc-db093c09c8a0-kube-api-access-tw64l\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv\" (UID: \"e5dde6d6-d08d-450b-8fcc-db093c09c8a0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"
Aug 01 00:15:59.781511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.781484    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"
Aug 01 00:15:59.913199 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:15:59.913140    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"]
Aug 01 00:15:59.916883 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:15:59.916852    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dde6d6_d08d_450b_8fcc_db093c09c8a0.slice/crio-21dbad4c0eb2c5b39aa3269f3f683863668b991dc183c6135a8de83893b5abd9 WatchSource:0}: Error finding container 21dbad4c0eb2c5b39aa3269f3f683863668b991dc183c6135a8de83893b5abd9: Status 404 returned error can't find the container with id 21dbad4c0eb2c5b39aa3269f3f683863668b991dc183c6135a8de83893b5abd9
Aug 01 00:16:00.577913 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:00.577876    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv" event={"ID":"e5dde6d6-d08d-450b-8fcc-db093c09c8a0","Type":"ContainerStarted","Data":"f6a84659fd708a44b4dacfde56eb0879fa140cb5a00e225aa86a72a887538e9f"}
Aug 01 00:16:00.578331 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:00.577954    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv" event={"ID":"e5dde6d6-d08d-450b-8fcc-db093c09c8a0","Type":"ContainerStarted","Data":"21dbad4c0eb2c5b39aa3269f3f683863668b991dc183c6135a8de83893b5abd9"}
Aug 01 00:16:00.578331 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:00.578061    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"
Aug 01 00:16:00.595509 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:00.595459    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv" podStartSLOduration=1.59544716 podStartE2EDuration="1.59544716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:16:00.593394377 +0000 UTC m=+1803.287441794" watchObservedRunningTime="2026-08-01 00:16:00.59544716 +0000 UTC m=+1803.289494577"
Aug 01 00:16:17.438977 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:17.438941    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:19.187138 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:19.187101    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"]
Aug 01 00:16:19.209076 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:19.209041    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:31.584652 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:31.584621    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"
Aug 01 00:16:34.206489 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.206444    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn" podUID="7d026bb6-05c1-41c0-8319-59655b15433d" containerName="main" containerID="cri-o://df83970e94a40a880a04537fff99b12511ccb951a79f11d424536c097f9c432f" gracePeriod=45
Aug 01 00:16:34.460427 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.460371    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:16:34.499120 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.499089    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-dshm\") pod \"7d026bb6-05c1-41c0-8319-59655b15433d\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") "
Aug 01 00:16:34.499288 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.499151    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-tmp-dir\") pod \"7d026bb6-05c1-41c0-8319-59655b15433d\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") "
Aug 01 00:16:34.499288 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.499180    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjcbn\" (UniqueName: \"kubernetes.io/projected/7d026bb6-05c1-41c0-8319-59655b15433d-kube-api-access-bjcbn\") pod \"7d026bb6-05c1-41c0-8319-59655b15433d\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") "
Aug 01 00:16:34.499288 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.499235    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-model-cache\") pod \"7d026bb6-05c1-41c0-8319-59655b15433d\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") "
Aug 01 00:16:34.499288 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.499264    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d026bb6-05c1-41c0-8319-59655b15433d-tls-certs\") pod \"7d026bb6-05c1-41c0-8319-59655b15433d\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") "
Aug 01 00:16:34.499512 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.499292    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-home\") pod \"7d026bb6-05c1-41c0-8319-59655b15433d\" (UID: \"7d026bb6-05c1-41c0-8319-59655b15433d\") "
Aug 01 00:16:34.499512 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.499423    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d026bb6-05c1-41c0-8319-59655b15433d" (UID: "7d026bb6-05c1-41c0-8319-59655b15433d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:16:34.499512 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.499494    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d026bb6-05c1-41c0-8319-59655b15433d" (UID: "7d026bb6-05c1-41c0-8319-59655b15433d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:16:34.499661 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.499629    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:16:34.499661 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.499651    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:16:34.499754 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.499655    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-home" (OuterVolumeSpecName: "home") pod "7d026bb6-05c1-41c0-8319-59655b15433d" (UID: "7d026bb6-05c1-41c0-8319-59655b15433d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:16:34.501555 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.501483    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-dshm" (OuterVolumeSpecName: "dshm") pod "7d026bb6-05c1-41c0-8319-59655b15433d" (UID: "7d026bb6-05c1-41c0-8319-59655b15433d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:16:34.501686 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.501581    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d026bb6-05c1-41c0-8319-59655b15433d-kube-api-access-bjcbn" (OuterVolumeSpecName: "kube-api-access-bjcbn") pod "7d026bb6-05c1-41c0-8319-59655b15433d" (UID: "7d026bb6-05c1-41c0-8319-59655b15433d"). InnerVolumeSpecName "kube-api-access-bjcbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:16:34.501854 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.501837    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d026bb6-05c1-41c0-8319-59655b15433d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d026bb6-05c1-41c0-8319-59655b15433d" (UID: "7d026bb6-05c1-41c0-8319-59655b15433d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:16:34.600590 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.600554    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d026bb6-05c1-41c0-8319-59655b15433d-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:16:34.600590 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.600584    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:16:34.600590 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.600596    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d026bb6-05c1-41c0-8319-59655b15433d-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:16:34.600822 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.600608    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjcbn\" (UniqueName: \"kubernetes.io/projected/7d026bb6-05c1-41c0-8319-59655b15433d-kube-api-access-bjcbn\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:16:34.723848 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.723812    2551 generic.go:356] "Generic (PLEG): container finished" podID="7d026bb6-05c1-41c0-8319-59655b15433d" containerID="df83970e94a40a880a04537fff99b12511ccb951a79f11d424536c097f9c432f" exitCode=0
Aug 01 00:16:34.724042 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.723857    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"
Aug 01 00:16:34.724042 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.723893    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn" event={"ID":"7d026bb6-05c1-41c0-8319-59655b15433d","Type":"ContainerDied","Data":"df83970e94a40a880a04537fff99b12511ccb951a79f11d424536c097f9c432f"}
Aug 01 00:16:34.724042 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.723978    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn" event={"ID":"7d026bb6-05c1-41c0-8319-59655b15433d","Type":"ContainerDied","Data":"021e672ab7d581bcc96f7bb9cc76f2c76adb213d47bc7398852fcf5ba97b3b18"}
Aug 01 00:16:34.724042 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.724007    2551 scope.go:117] "RemoveContainer" containerID="df83970e94a40a880a04537fff99b12511ccb951a79f11d424536c097f9c432f"
Aug 01 00:16:34.733839 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.733821    2551 scope.go:117] "RemoveContainer" containerID="df83970e94a40a880a04537fff99b12511ccb951a79f11d424536c097f9c432f"
Aug 01 00:16:34.734085 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:16:34.734064    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df83970e94a40a880a04537fff99b12511ccb951a79f11d424536c097f9c432f\": container with ID starting with df83970e94a40a880a04537fff99b12511ccb951a79f11d424536c097f9c432f not found: ID does not exist" containerID="df83970e94a40a880a04537fff99b12511ccb951a79f11d424536c097f9c432f"
Aug 01 00:16:34.734151 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.734096    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df83970e94a40a880a04537fff99b12511ccb951a79f11d424536c097f9c432f"} err="failed to get container status \"df83970e94a40a880a04537fff99b12511ccb951a79f11d424536c097f9c432f\": rpc error: code = NotFound desc = could not find container \"df83970e94a40a880a04537fff99b12511ccb951a79f11d424536c097f9c432f\": container with ID starting with df83970e94a40a880a04537fff99b12511ccb951a79f11d424536c097f9c432f not found: ID does not exist"
Aug 01 00:16:34.743866 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.743844    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"]
Aug 01 00:16:34.746890 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:34.746871    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-xntgn"]
Aug 01 00:16:35.547556 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:35.547523    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:35.649195 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:35.649163    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:35.747819 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:35.747787    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:35.947239 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:35.947165    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d026bb6-05c1-41c0-8319-59655b15433d" path="/var/lib/kubelet/pods/7d026bb6-05c1-41c0-8319-59655b15433d/volumes"
Aug 01 00:16:40.573325 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:40.573291    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:42.963336 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:42.963307    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"]
Aug 01 00:16:42.963704 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:42.963618    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d026bb6-05c1-41c0-8319-59655b15433d" containerName="main"
Aug 01 00:16:42.963704 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:42.963670    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d026bb6-05c1-41c0-8319-59655b15433d" containerName="main"
Aug 01 00:16:42.963778 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:42.963726    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d026bb6-05c1-41c0-8319-59655b15433d" containerName="main"
Aug 01 00:16:42.967945 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:42.967903    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:42.970230 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:42.970202    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Aug 01 00:16:42.977026 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:42.977004    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"]
Aug 01 00:16:42.995563 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:42.995536    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:43.075218 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.075177    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.075386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.075239    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73dc5ed4-3314-4a43-8743-ead6da377acb-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.075386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.075294    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxdr\" (UniqueName: \"kubernetes.io/projected/73dc5ed4-3314-4a43-8743-ead6da377acb-kube-api-access-jcxdr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.075386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.075312    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.075386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.075354    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.075526 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.075386    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.176654 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.176624    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73dc5ed4-3314-4a43-8743-ead6da377acb-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.176839 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.176687    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxdr\" (UniqueName: \"kubernetes.io/projected/73dc5ed4-3314-4a43-8743-ead6da377acb-kube-api-access-jcxdr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.176839 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.176716    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.176839 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.176744    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.176839 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.176768    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.176839 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.176823    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.177235 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.177211    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.177299 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.177233    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.177299 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.177259    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.179009 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.178988    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.179307 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.179287    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73dc5ed4-3314-4a43-8743-ead6da377acb-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.187050 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.187021    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxdr\" (UniqueName: \"kubernetes.io/projected/73dc5ed4-3314-4a43-8743-ead6da377acb-kube-api-access-jcxdr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.198969 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.198943    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:43.278539 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.278504    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:43.410324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.410281    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"]
Aug 01 00:16:43.413328 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:16:43.413293    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73dc5ed4_3314_4a43_8743_ead6da377acb.slice/crio-5118e3addebaac6b81debaf83ea96470015ba33282fb6dd1efb4545b81cb5331 WatchSource:0}: Error finding container 5118e3addebaac6b81debaf83ea96470015ba33282fb6dd1efb4545b81cb5331: Status 404 returned error can't find the container with id 5118e3addebaac6b81debaf83ea96470015ba33282fb6dd1efb4545b81cb5331
Aug 01 00:16:43.762944 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.762903    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n" event={"ID":"73dc5ed4-3314-4a43-8743-ead6da377acb","Type":"ContainerStarted","Data":"e52a93311776a2dd99d5e1a92b612ae0e29b02fbf8c8b2ed95713b6f935eef66"}
Aug 01 00:16:43.763108 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.762961    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n" event={"ID":"73dc5ed4-3314-4a43-8743-ead6da377acb","Type":"ContainerStarted","Data":"5118e3addebaac6b81debaf83ea96470015ba33282fb6dd1efb4545b81cb5331"}
Aug 01 00:16:43.781044 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:43.780972    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n" podStartSLOduration=1.78095599 podStartE2EDuration="1.78095599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:16:43.779337347 +0000 UTC m=+1846.473384764" watchObservedRunningTime="2026-08-01 00:16:43.78095599 +0000 UTC m=+1846.475003407"
Aug 01 00:16:44.279311 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:44.279275    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:45.863621 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:45.863592    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:53.279035 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:53.279001    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:53.291846 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:53.291814    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:53.821046 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:53.821018    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:16:57.282499 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:57.282463    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:58.808926 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:16:58.808894    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:04.277280 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:04.277233    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:04.376670 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:04.376631    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:07.403347 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:07.403318    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:16.626669 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:16.626626    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:35.669145 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:35.669062    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"]
Aug 01 00:17:35.672319 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:35.672286    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:35.678298 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:17:35.678263    2551 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 01 00:17:35.678298 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:17:35.678298 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:17:35.678298 ip-10-0-131-136 kubenswrapper[2551]:  > containerID="e52a93311776a2dd99d5e1a92b612ae0e29b02fbf8c8b2ed95713b6f935eef66" cmd=["/bin/sleep","15"]
Aug 01 00:17:35.678606 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:35.678318    2551 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n" message="" err=<
Aug 01 00:17:35.678606 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:17:35.678606 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:17:35.678606 ip-10-0-131-136 kubenswrapper[2551]:  >
Aug 01 00:17:35.678606 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:17:35.678333    2551 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 01 00:17:35.678606 ip-10-0-131-136 kubenswrapper[2551]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 01 00:17:35.678606 ip-10-0-131-136 kubenswrapper[2551]:         , stdout: , stderr: , exit code -1
Aug 01 00:17:35.678606 ip-10-0-131-136 kubenswrapper[2551]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n" podUID="73dc5ed4-3314-4a43-8743-ead6da377acb" containerName="main" containerID="cri-o://e52a93311776a2dd99d5e1a92b612ae0e29b02fbf8c8b2ed95713b6f935eef66"
Aug 01 00:17:35.678606 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:35.678388    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n" podUID="73dc5ed4-3314-4a43-8743-ead6da377acb" containerName="main" containerID="cri-o://e52a93311776a2dd99d5e1a92b612ae0e29b02fbf8c8b2ed95713b6f935eef66" gracePeriod=60
Aug 01 00:17:35.770481 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:35.770418    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:35.810246 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:35.810214    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n" podUID="73dc5ed4-3314-4a43-8743-ead6da377acb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Aug 01 00:17:35.990286 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:35.990254    2551 generic.go:356] "Generic (PLEG): container finished" podID="73dc5ed4-3314-4a43-8743-ead6da377acb" containerID="e52a93311776a2dd99d5e1a92b612ae0e29b02fbf8c8b2ed95713b6f935eef66" exitCode=0
Aug 01 00:17:35.990464 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:35.990291    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n" event={"ID":"73dc5ed4-3314-4a43-8743-ead6da377acb","Type":"ContainerDied","Data":"e52a93311776a2dd99d5e1a92b612ae0e29b02fbf8c8b2ed95713b6f935eef66"}
Aug 01 00:17:36.025019 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.024993    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:17:36.144084 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.144055    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-model-cache\") pod \"73dc5ed4-3314-4a43-8743-ead6da377acb\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") "
Aug 01 00:17:36.144084 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.144086    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-dshm\") pod \"73dc5ed4-3314-4a43-8743-ead6da377acb\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") "
Aug 01 00:17:36.144327 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.144112    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcxdr\" (UniqueName: \"kubernetes.io/projected/73dc5ed4-3314-4a43-8743-ead6da377acb-kube-api-access-jcxdr\") pod \"73dc5ed4-3314-4a43-8743-ead6da377acb\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") "
Aug 01 00:17:36.144327 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.144133    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73dc5ed4-3314-4a43-8743-ead6da377acb-tls-certs\") pod \"73dc5ed4-3314-4a43-8743-ead6da377acb\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") "
Aug 01 00:17:36.144327 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.144215    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-tmp-dir\") pod \"73dc5ed4-3314-4a43-8743-ead6da377acb\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") "
Aug 01 00:17:36.144327 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.144260    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-home\") pod \"73dc5ed4-3314-4a43-8743-ead6da377acb\" (UID: \"73dc5ed4-3314-4a43-8743-ead6da377acb\") "
Aug 01 00:17:36.144546 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.144385    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-model-cache" (OuterVolumeSpecName: "model-cache") pod "73dc5ed4-3314-4a43-8743-ead6da377acb" (UID: "73dc5ed4-3314-4a43-8743-ead6da377acb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:17:36.144546 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.144454    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73dc5ed4-3314-4a43-8743-ead6da377acb" (UID: "73dc5ed4-3314-4a43-8743-ead6da377acb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:17:36.144546 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.144499    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-home" (OuterVolumeSpecName: "home") pod "73dc5ed4-3314-4a43-8743-ead6da377acb" (UID: "73dc5ed4-3314-4a43-8743-ead6da377acb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:17:36.144660 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.144575    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:17:36.144660 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.144591    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:17:36.144660 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.144605    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:17:36.146401 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.146373    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73dc5ed4-3314-4a43-8743-ead6da377acb-kube-api-access-jcxdr" (OuterVolumeSpecName: "kube-api-access-jcxdr") pod "73dc5ed4-3314-4a43-8743-ead6da377acb" (UID: "73dc5ed4-3314-4a43-8743-ead6da377acb"). InnerVolumeSpecName "kube-api-access-jcxdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:17:36.146401 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.146387    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-dshm" (OuterVolumeSpecName: "dshm") pod "73dc5ed4-3314-4a43-8743-ead6da377acb" (UID: "73dc5ed4-3314-4a43-8743-ead6da377acb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:17:36.146549 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.146454    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73dc5ed4-3314-4a43-8743-ead6da377acb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73dc5ed4-3314-4a43-8743-ead6da377acb" (UID: "73dc5ed4-3314-4a43-8743-ead6da377acb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:17:36.245971 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.245945    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73dc5ed4-3314-4a43-8743-ead6da377acb-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:17:36.246100 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.245973    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcxdr\" (UniqueName: \"kubernetes.io/projected/73dc5ed4-3314-4a43-8743-ead6da377acb-kube-api-access-jcxdr\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:17:36.246100 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.245983    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73dc5ed4-3314-4a43-8743-ead6da377acb-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:17:36.995612 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.995577    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n" event={"ID":"73dc5ed4-3314-4a43-8743-ead6da377acb","Type":"ContainerDied","Data":"5118e3addebaac6b81debaf83ea96470015ba33282fb6dd1efb4545b81cb5331"}
Aug 01 00:17:36.996035 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.995624    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"
Aug 01 00:17:36.996035 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:36.995640    2551 scope.go:117] "RemoveContainer" containerID="e52a93311776a2dd99d5e1a92b612ae0e29b02fbf8c8b2ed95713b6f935eef66"
Aug 01 00:17:37.022403 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:37.022376    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"]
Aug 01 00:17:37.027811 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:37.027789    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952447n"]
Aug 01 00:17:37.946592 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:37.946558    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73dc5ed4-3314-4a43-8743-ead6da377acb" path="/var/lib/kubelet/pods/73dc5ed4-3314-4a43-8743-ead6da377acb/volumes"
Aug 01 00:17:40.521909 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:40.521871    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:40.623981 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:40.623911    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:40.724742 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:40.724710    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:42.762071 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:42.762023    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:45.595711 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:45.595676    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:48.958360 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:48.958326    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"]
Aug 01 00:17:48.958892 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:48.958713    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73dc5ed4-3314-4a43-8743-ead6da377acb" containerName="main"
Aug 01 00:17:48.958892 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:48.958729    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="73dc5ed4-3314-4a43-8743-ead6da377acb" containerName="main"
Aug 01 00:17:48.958892 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:48.958838    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="73dc5ed4-3314-4a43-8743-ead6da377acb" containerName="main"
Aug 01 00:17:48.962979 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:48.962952    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:48.965743 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:48.965714    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-zcnn2\""
Aug 01 00:17:48.965889 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:48.965754    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Aug 01 00:17:48.965889 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:48.965714    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Aug 01 00:17:48.965889 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:48.965729    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Aug 01 00:17:48.970889 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:48.970870    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"]
Aug 01 00:17:48.986452 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:48.986418    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:49.061024 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.060986    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhmfk\" (UniqueName: \"kubernetes.io/projected/ae1c5867-4578-4cf7-bc85-0790f923e4de-kube-api-access-lhmfk\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.061217 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.061034    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.061217 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.061057    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae1c5867-4578-4cf7-bc85-0790f923e4de-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.061217 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.061137    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-home\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.061217 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.061199    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.061378 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.061258    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-dshm\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.162409 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.162367    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.162598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.162431    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-dshm\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.162598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.162457    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhmfk\" (UniqueName: \"kubernetes.io/projected/ae1c5867-4578-4cf7-bc85-0790f923e4de-kube-api-access-lhmfk\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.162598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.162480    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.162598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.162495    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae1c5867-4578-4cf7-bc85-0790f923e4de-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.162598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.162529    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-home\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.162870 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.162824    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.162962 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.162882    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.162962 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.162895    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-home\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.164828 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.164802    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-dshm\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.165029 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.165009    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae1c5867-4578-4cf7-bc85-0790f923e4de-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.170089 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.170069    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhmfk\" (UniqueName: \"kubernetes.io/projected/ae1c5867-4578-4cf7-bc85-0790f923e4de-kube-api-access-lhmfk\") pod \"tri-v1-kserve-79f8f98bc5-qrl4w\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.274097 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.274066    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:49.405831 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:49.405806    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"]
Aug 01 00:17:49.408162 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:17:49.408131    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1c5867_4578_4cf7_bc85_0790f923e4de.slice/crio-acf1665fc27166927149eefc3ab2c2d4070612a73193072a74aab0a01091f01a WatchSource:0}: Error finding container acf1665fc27166927149eefc3ab2c2d4070612a73193072a74aab0a01091f01a: Status 404 returned error can't find the container with id acf1665fc27166927149eefc3ab2c2d4070612a73193072a74aab0a01091f01a
Aug 01 00:17:50.057878 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:50.057760    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w" event={"ID":"ae1c5867-4578-4cf7-bc85-0790f923e4de","Type":"ContainerStarted","Data":"117ff5d8a0bf737692160f4526b784c5345a50604a172f7bd17d84dd0482c686"}
Aug 01 00:17:50.057878 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:50.057836    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w" event={"ID":"ae1c5867-4578-4cf7-bc85-0790f923e4de","Type":"ContainerStarted","Data":"acf1665fc27166927149eefc3ab2c2d4070612a73193072a74aab0a01091f01a"}
Aug 01 00:17:50.073069 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:50.073017    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w" podStartSLOduration=2.072999081 podStartE2EDuration="2.072999081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:17:50.071853282 +0000 UTC m=+1912.765900698" watchObservedRunningTime="2026-08-01 00:17:50.072999081 +0000 UTC m=+1912.767046499"
Aug 01 00:17:50.274569 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:50.274528    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:58.336874 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:58.336789    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:59.275129 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:59.275092    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:17:59.287558 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:17:59.287529    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:18:00.112543 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:00.112513    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:18:12.559274 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.559232    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"]
Aug 01 00:18:12.563038 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.563012    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.572771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.572736    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Aug 01 00:18:12.575340 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.575306    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.575608 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.575550    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-dshm\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.575735 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.575658    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-model-cache\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.575815 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.575742    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-home\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.575815 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.575768    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.575815 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.575794    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7sck\" (UniqueName: \"kubernetes.io/projected/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-kube-api-access-l7sck\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.576846 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.576818    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"]
Aug 01 00:18:12.594568 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.594535    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:12.676334 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.676290    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-home\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.676537 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.676341    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.676537 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.676368    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7sck\" (UniqueName: \"kubernetes.io/projected/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-kube-api-access-l7sck\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.676537 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.676421    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.676537 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.676442    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-dshm\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.676789 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.676576    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-model-cache\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.676789 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.676723    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-home\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.676789 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.676777    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.676943 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.676888    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-model-cache\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.678687 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.678666    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-dshm\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.679011 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.678992    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.683948 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.683908    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7sck\" (UniqueName: \"kubernetes.io/projected/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-kube-api-access-l7sck\") pod \"tri-v2-kserve-dcb554bf5-hwqvc\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:12.885835 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:12.885724    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:13.016114 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:13.016085    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"]
Aug 01 00:18:13.019051 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:18:13.019024    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ba5ce4_b0e2_4086_aab8_1078de4fc30e.slice/crio-76568b43fe452e149435e12b879c0856a09c2d7065e11ef7f3db4f06c83db471 WatchSource:0}: Error finding container 76568b43fe452e149435e12b879c0856a09c2d7065e11ef7f3db4f06c83db471: Status 404 returned error can't find the container with id 76568b43fe452e149435e12b879c0856a09c2d7065e11ef7f3db4f06c83db471
Aug 01 00:18:13.155731 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:13.155640    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc" event={"ID":"09ba5ce4-b0e2-4086-aab8-1078de4fc30e","Type":"ContainerStarted","Data":"33c4224b2836eeb5451b39619789dfc6edcfcb176cb9cfaefe57a588106c563c"}
Aug 01 00:18:13.155731 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:13.155706    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc" event={"ID":"09ba5ce4-b0e2-4086-aab8-1078de4fc30e","Type":"ContainerStarted","Data":"76568b43fe452e149435e12b879c0856a09c2d7065e11ef7f3db4f06c83db471"}
Aug 01 00:18:13.171590 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:13.171535    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc" podStartSLOduration=1.171499899 podStartE2EDuration="1.171499899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:18:13.170625114 +0000 UTC m=+1935.864672532" watchObservedRunningTime="2026-08-01 00:18:13.171499899 +0000 UTC m=+1935.865547314"
Aug 01 00:18:13.886501 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:13.886454    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:22.503476 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:22.503440    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:22.829141 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:22.829050    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:22.886478 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:22.886436    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:22.898951 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:22.898906    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:23.212062 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:23.211979    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:18:32.679047 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:32.679006    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:34.969885 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:34.969843    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"]
Aug 01 00:18:34.989102 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:34.989073    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:35.084649 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:35.084609    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:36.757875 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:36.757833    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:36.858482 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:36.858446    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:41.584492 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:41.584452    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv" podUID="e5dde6d6-d08d-450b-8fcc-db093c09c8a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:18:49.989401 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:49.989321    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv" podUID="e5dde6d6-d08d-450b-8fcc-db093c09c8a0" containerName="main" containerID="cri-o://f6a84659fd708a44b4dacfde56eb0879fa140cb5a00e225aa86a72a887538e9f" gracePeriod=45
Aug 01 00:18:50.240741 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.240667    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"
Aug 01 00:18:50.323480 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.323446    2551 generic.go:356] "Generic (PLEG): container finished" podID="e5dde6d6-d08d-450b-8fcc-db093c09c8a0" containerID="f6a84659fd708a44b4dacfde56eb0879fa140cb5a00e225aa86a72a887538e9f" exitCode=0
Aug 01 00:18:50.323652 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.323493    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"
Aug 01 00:18:50.323652 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.323497    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv" event={"ID":"e5dde6d6-d08d-450b-8fcc-db093c09c8a0","Type":"ContainerDied","Data":"f6a84659fd708a44b4dacfde56eb0879fa140cb5a00e225aa86a72a887538e9f"}
Aug 01 00:18:50.323652 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.323631    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv" event={"ID":"e5dde6d6-d08d-450b-8fcc-db093c09c8a0","Type":"ContainerDied","Data":"21dbad4c0eb2c5b39aa3269f3f683863668b991dc183c6135a8de83893b5abd9"}
Aug 01 00:18:50.323758 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.323660    2551 scope.go:117] "RemoveContainer" containerID="f6a84659fd708a44b4dacfde56eb0879fa140cb5a00e225aa86a72a887538e9f"
Aug 01 00:18:50.331804 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.331783    2551 scope.go:117] "RemoveContainer" containerID="f6a84659fd708a44b4dacfde56eb0879fa140cb5a00e225aa86a72a887538e9f"
Aug 01 00:18:50.332091 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:18:50.332071    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a84659fd708a44b4dacfde56eb0879fa140cb5a00e225aa86a72a887538e9f\": container with ID starting with f6a84659fd708a44b4dacfde56eb0879fa140cb5a00e225aa86a72a887538e9f not found: ID does not exist" containerID="f6a84659fd708a44b4dacfde56eb0879fa140cb5a00e225aa86a72a887538e9f"
Aug 01 00:18:50.332157 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.332105    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a84659fd708a44b4dacfde56eb0879fa140cb5a00e225aa86a72a887538e9f"} err="failed to get container status \"f6a84659fd708a44b4dacfde56eb0879fa140cb5a00e225aa86a72a887538e9f\": rpc error: code = NotFound desc = could not find container \"f6a84659fd708a44b4dacfde56eb0879fa140cb5a00e225aa86a72a887538e9f\": container with ID starting with f6a84659fd708a44b4dacfde56eb0879fa140cb5a00e225aa86a72a887538e9f not found: ID does not exist"
Aug 01 00:18:50.426656 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.426622    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5dde6d6-d08d-450b-8fcc-db093c09c8a0-tls-certs\") pod \"e5dde6d6-d08d-450b-8fcc-db093c09c8a0\" (UID: \"e5dde6d6-d08d-450b-8fcc-db093c09c8a0\") "
Aug 01 00:18:50.426656 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.426662    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw64l\" (UniqueName: \"kubernetes.io/projected/e5dde6d6-d08d-450b-8fcc-db093c09c8a0-kube-api-access-tw64l\") pod \"e5dde6d6-d08d-450b-8fcc-db093c09c8a0\" (UID: \"e5dde6d6-d08d-450b-8fcc-db093c09c8a0\") "
Aug 01 00:18:50.428744 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.428714    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5dde6d6-d08d-450b-8fcc-db093c09c8a0-kube-api-access-tw64l" (OuterVolumeSpecName: "kube-api-access-tw64l") pod "e5dde6d6-d08d-450b-8fcc-db093c09c8a0" (UID: "e5dde6d6-d08d-450b-8fcc-db093c09c8a0"). InnerVolumeSpecName "kube-api-access-tw64l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:18:50.428907 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.428894    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5dde6d6-d08d-450b-8fcc-db093c09c8a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5dde6d6-d08d-450b-8fcc-db093c09c8a0" (UID: "e5dde6d6-d08d-450b-8fcc-db093c09c8a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:18:50.527720 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.527634    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5dde6d6-d08d-450b-8fcc-db093c09c8a0-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:18:50.527720 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.527663    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw64l\" (UniqueName: \"kubernetes.io/projected/e5dde6d6-d08d-450b-8fcc-db093c09c8a0-kube-api-access-tw64l\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:18:50.644622 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.644588    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"]
Aug 01 00:18:50.649957 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:50.649933    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7wdcv"]
Aug 01 00:18:51.945988 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:51.945952    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5dde6d6-d08d-450b-8fcc-db093c09c8a0" path="/var/lib/kubelet/pods/e5dde6d6-d08d-450b-8fcc-db093c09c8a0/volumes"
Aug 01 00:18:54.776533 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:54.776494    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:54.875943 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:18:54.875865    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:19:00.870210 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:00.870176    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:19:06.306322 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:06.306284    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:19:06.366009 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:06.365960    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:19:39.093298 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.093268    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:19:39.264627 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.264596    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"]
Aug 01 00:19:39.264960 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.264946    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5dde6d6-d08d-450b-8fcc-db093c09c8a0" containerName="main"
Aug 01 00:19:39.265023 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.264969    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5dde6d6-d08d-450b-8fcc-db093c09c8a0" containerName="main"
Aug 01 00:19:39.265060 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.265039    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5dde6d6-d08d-450b-8fcc-db093c09c8a0" containerName="main"
Aug 01 00:19:39.268014 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.267991    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"
Aug 01 00:19:39.270741 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.270710    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-dlcqx\""
Aug 01 00:19:39.270875 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.270745    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Aug 01 00:19:39.270875 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.270818    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-76kns\""
Aug 01 00:19:39.271026 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.270930    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Aug 01 00:19:39.271026 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.271015    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Aug 01 00:19:39.282409 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.282369    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"]
Aug 01 00:19:39.310099 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.310064    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:19:39.329754 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.329721    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dppl7\" (UniqueName: \"kubernetes.io/projected/eef1d222-953c-41c5-9ab6-92e9a4ef4c1f-kube-api-access-dppl7\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl\" (UID: \"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"
Aug 01 00:19:39.329944 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.329875    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eef1d222-953c-41c5-9ab6-92e9a4ef4c1f-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl\" (UID: \"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"
Aug 01 00:19:39.430430 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.430330    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eef1d222-953c-41c5-9ab6-92e9a4ef4c1f-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl\" (UID: \"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"
Aug 01 00:19:39.430430 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.430390    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dppl7\" (UniqueName: \"kubernetes.io/projected/eef1d222-953c-41c5-9ab6-92e9a4ef4c1f-kube-api-access-dppl7\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl\" (UID: \"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"
Aug 01 00:19:39.433088 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.433055    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eef1d222-953c-41c5-9ab6-92e9a4ef4c1f-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl\" (UID: \"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"
Aug 01 00:19:39.437731 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.437703    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dppl7\" (UniqueName: \"kubernetes.io/projected/eef1d222-953c-41c5-9ab6-92e9a4ef4c1f-kube-api-access-dppl7\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl\" (UID: \"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"
Aug 01 00:19:39.580982 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.580941    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"
Aug 01 00:19:39.709514 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:39.709489    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"]
Aug 01 00:19:39.711638 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:19:39.711610    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef1d222_953c_41c5_9ab6_92e9a4ef4c1f.slice/crio-666594fe77994a90d9f28d94f903571e223ab3fa15eda7760b91e6d2484f44bd WatchSource:0}: Error finding container 666594fe77994a90d9f28d94f903571e223ab3fa15eda7760b91e6d2484f44bd: Status 404 returned error can't find the container with id 666594fe77994a90d9f28d94f903571e223ab3fa15eda7760b91e6d2484f44bd
Aug 01 00:19:40.547550 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:40.547507    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl" event={"ID":"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f","Type":"ContainerStarted","Data":"234393eb1662ea5448c49ea687dc32a210aef6cf940a40cf8724183cfa57906e"}
Aug 01 00:19:40.548061 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:40.547616    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl" event={"ID":"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f","Type":"ContainerStarted","Data":"666594fe77994a90d9f28d94f903571e223ab3fa15eda7760b91e6d2484f44bd"}
Aug 01 00:19:40.548061 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:40.547729    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"
Aug 01 00:19:40.563730 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:40.563670    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl" podStartSLOduration=1.56365037 podStartE2EDuration="1.56365037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:19:40.561583642 +0000 UTC m=+2023.255631072" watchObservedRunningTime="2026-08-01 00:19:40.56365037 +0000 UTC m=+2023.257697787"
Aug 01 00:19:46.540503 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:46.540469    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:19:48.146263 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:19:48.146229    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:20:11.554620 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:11.554588    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"
Aug 01 00:20:18.668188 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:18.668136    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:20:18.770622 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:18.770580    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:20:18.872432 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:18.872394    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:20:18.880013 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:18.879979    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"]
Aug 01 00:20:18.883575 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:18.883550    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"
Aug 01 00:20:18.886260 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:18.886233    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-tr8wk\""
Aug 01 00:20:18.886409 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:18.886235    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Aug 01 00:20:18.894688 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:18.894657    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"]
Aug 01 00:20:18.969272 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:18.969215    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:20:18.997422 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:18.997377    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twstz\" (UniqueName: \"kubernetes.io/projected/909344cf-c3c5-4034-93cf-500f101a32e8-kube-api-access-twstz\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8\" (UID: \"909344cf-c3c5-4034-93cf-500f101a32e8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"
Aug 01 00:20:18.997618 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:18.997436    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909344cf-c3c5-4034-93cf-500f101a32e8-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8\" (UID: \"909344cf-c3c5-4034-93cf-500f101a32e8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"
Aug 01 00:20:19.098491 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:19.098448    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twstz\" (UniqueName: \"kubernetes.io/projected/909344cf-c3c5-4034-93cf-500f101a32e8-kube-api-access-twstz\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8\" (UID: \"909344cf-c3c5-4034-93cf-500f101a32e8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"
Aug 01 00:20:19.098491 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:19.098499    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909344cf-c3c5-4034-93cf-500f101a32e8-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8\" (UID: \"909344cf-c3c5-4034-93cf-500f101a32e8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"
Aug 01 00:20:19.101457 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:19.101426    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909344cf-c3c5-4034-93cf-500f101a32e8-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8\" (UID: \"909344cf-c3c5-4034-93cf-500f101a32e8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"
Aug 01 00:20:19.108748 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:19.108717    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twstz\" (UniqueName: \"kubernetes.io/projected/909344cf-c3c5-4034-93cf-500f101a32e8-kube-api-access-twstz\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8\" (UID: \"909344cf-c3c5-4034-93cf-500f101a32e8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"
Aug 01 00:20:19.197286 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:19.197241    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"
Aug 01 00:20:19.351535 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:19.351507    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"]
Aug 01 00:20:19.353769 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:20:19.353733    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909344cf_c3c5_4034_93cf_500f101a32e8.slice/crio-cca1ea913aec685b493878d81b249724e85c68d39ff9ee252103c4c58f0027b1 WatchSource:0}: Error finding container cca1ea913aec685b493878d81b249724e85c68d39ff9ee252103c4c58f0027b1: Status 404 returned error can't find the container with id cca1ea913aec685b493878d81b249724e85c68d39ff9ee252103c4c58f0027b1
Aug 01 00:20:19.732341 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:19.732306    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8" event={"ID":"909344cf-c3c5-4034-93cf-500f101a32e8","Type":"ContainerStarted","Data":"9a89305733c7afe0790d84183767da3d8327b187fe8df56060ae25c65f694b7a"}
Aug 01 00:20:19.732724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:19.732360    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8" event={"ID":"909344cf-c3c5-4034-93cf-500f101a32e8","Type":"ContainerStarted","Data":"cca1ea913aec685b493878d81b249724e85c68d39ff9ee252103c4c58f0027b1"}
Aug 01 00:20:19.732724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:19.732410    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"
Aug 01 00:20:19.748613 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:19.748548    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8" podStartSLOduration=1.748527183 podStartE2EDuration="1.748527183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:20:19.747543992 +0000 UTC m=+2062.441591408" watchObservedRunningTime="2026-08-01 00:20:19.748527183 +0000 UTC m=+2062.442574599"
Aug 01 00:20:22.341164 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:22.341132    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"]
Aug 01 00:20:22.370282 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:22.370246    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:20:23.138073 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:23.138031    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:20:23.708620 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:23.708581    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"]
Aug 01 00:20:23.802281 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:23.802241    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:20:25.700129 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:25.700079    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:20:37.361123 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.361076    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc" podUID="09ba5ce4-b0e2-4086-aab8-1078de4fc30e" containerName="main" containerID="cri-o://33c4224b2836eeb5451b39619789dfc6edcfcb176cb9cfaefe57a588106c563c" gracePeriod=45
Aug 01 00:20:37.602538 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.602512    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:20:37.669982 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.669889    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-model-cache\") pod \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") "
Aug 01 00:20:37.669982 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.669955    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7sck\" (UniqueName: \"kubernetes.io/projected/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-kube-api-access-l7sck\") pod \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") "
Aug 01 00:20:37.670208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.669991    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-tmp-dir\") pod \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") "
Aug 01 00:20:37.670208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.670011    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-tls-certs\") pod \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") "
Aug 01 00:20:37.670208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.670048    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-dshm\") pod \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") "
Aug 01 00:20:37.670208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.670070    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-home\") pod \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\" (UID: \"09ba5ce4-b0e2-4086-aab8-1078de4fc30e\") "
Aug 01 00:20:37.670414 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.670180    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-model-cache" (OuterVolumeSpecName: "model-cache") pod "09ba5ce4-b0e2-4086-aab8-1078de4fc30e" (UID: "09ba5ce4-b0e2-4086-aab8-1078de4fc30e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:20:37.670414 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.670282    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09ba5ce4-b0e2-4086-aab8-1078de4fc30e" (UID: "09ba5ce4-b0e2-4086-aab8-1078de4fc30e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:20:37.670414 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.670344    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:20:37.670565 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.670433    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-home" (OuterVolumeSpecName: "home") pod "09ba5ce4-b0e2-4086-aab8-1078de4fc30e" (UID: "09ba5ce4-b0e2-4086-aab8-1078de4fc30e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:20:37.672349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.672323    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-dshm" (OuterVolumeSpecName: "dshm") pod "09ba5ce4-b0e2-4086-aab8-1078de4fc30e" (UID: "09ba5ce4-b0e2-4086-aab8-1078de4fc30e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:20:37.672457 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.672367    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09ba5ce4-b0e2-4086-aab8-1078de4fc30e" (UID: "09ba5ce4-b0e2-4086-aab8-1078de4fc30e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:20:37.672457 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.672399    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-kube-api-access-l7sck" (OuterVolumeSpecName: "kube-api-access-l7sck") pod "09ba5ce4-b0e2-4086-aab8-1078de4fc30e" (UID: "09ba5ce4-b0e2-4086-aab8-1078de4fc30e"). InnerVolumeSpecName "kube-api-access-l7sck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:20:37.770771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.770733    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:20:37.770771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.770764    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:20:37.770771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.770772    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:20:37.770771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.770780    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:20:37.771069 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.770789    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7sck\" (UniqueName: \"kubernetes.io/projected/09ba5ce4-b0e2-4086-aab8-1078de4fc30e-kube-api-access-l7sck\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:20:37.818733 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.818704    2551 generic.go:356] "Generic (PLEG): container finished" podID="09ba5ce4-b0e2-4086-aab8-1078de4fc30e" containerID="33c4224b2836eeb5451b39619789dfc6edcfcb176cb9cfaefe57a588106c563c" exitCode=0
Aug 01 00:20:37.818891 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.818744    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc" event={"ID":"09ba5ce4-b0e2-4086-aab8-1078de4fc30e","Type":"ContainerDied","Data":"33c4224b2836eeb5451b39619789dfc6edcfcb176cb9cfaefe57a588106c563c"}
Aug 01 00:20:37.818891 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.818754    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"
Aug 01 00:20:37.818891 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.818814    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc" event={"ID":"09ba5ce4-b0e2-4086-aab8-1078de4fc30e","Type":"ContainerDied","Data":"76568b43fe452e149435e12b879c0856a09c2d7065e11ef7f3db4f06c83db471"}
Aug 01 00:20:37.818891 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.818837    2551 scope.go:117] "RemoveContainer" containerID="33c4224b2836eeb5451b39619789dfc6edcfcb176cb9cfaefe57a588106c563c"
Aug 01 00:20:37.827829 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.827810    2551 scope.go:117] "RemoveContainer" containerID="33c4224b2836eeb5451b39619789dfc6edcfcb176cb9cfaefe57a588106c563c"
Aug 01 00:20:37.828128 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:20:37.828108    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c4224b2836eeb5451b39619789dfc6edcfcb176cb9cfaefe57a588106c563c\": container with ID starting with 33c4224b2836eeb5451b39619789dfc6edcfcb176cb9cfaefe57a588106c563c not found: ID does not exist" containerID="33c4224b2836eeb5451b39619789dfc6edcfcb176cb9cfaefe57a588106c563c"
Aug 01 00:20:37.828203 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.828136    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c4224b2836eeb5451b39619789dfc6edcfcb176cb9cfaefe57a588106c563c"} err="failed to get container status \"33c4224b2836eeb5451b39619789dfc6edcfcb176cb9cfaefe57a588106c563c\": rpc error: code = NotFound desc = could not find container \"33c4224b2836eeb5451b39619789dfc6edcfcb176cb9cfaefe57a588106c563c\": container with ID starting with 33c4224b2836eeb5451b39619789dfc6edcfcb176cb9cfaefe57a588106c563c not found: ID does not exist"
Aug 01 00:20:37.838798 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.838775    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"]
Aug 01 00:20:37.844786 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.844765    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-hwqvc"]
Aug 01 00:20:37.946586 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:37.946517    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ba5ce4-b0e2-4086-aab8-1078de4fc30e" path="/var/lib/kubelet/pods/09ba5ce4-b0e2-4086-aab8-1078de4fc30e/volumes"
Aug 01 00:20:38.727610 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.727543    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w" podUID="ae1c5867-4578-4cf7-bc85-0790f923e4de" containerName="main" containerID="cri-o://117ff5d8a0bf737692160f4526b784c5345a50604a172f7bd17d84dd0482c686" gracePeriod=45
Aug 01 00:20:38.968658 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.968634    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:20:38.978519 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.978460    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae1c5867-4578-4cf7-bc85-0790f923e4de-tls-certs\") pod \"ae1c5867-4578-4cf7-bc85-0790f923e4de\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") "
Aug 01 00:20:38.978648 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.978516    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-dshm\") pod \"ae1c5867-4578-4cf7-bc85-0790f923e4de\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") "
Aug 01 00:20:38.978648 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.978611    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-model-cache\") pod \"ae1c5867-4578-4cf7-bc85-0790f923e4de\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") "
Aug 01 00:20:38.978768 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.978663    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-home\") pod \"ae1c5867-4578-4cf7-bc85-0790f923e4de\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") "
Aug 01 00:20:38.978768 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.978687    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-tmp-dir\") pod \"ae1c5867-4578-4cf7-bc85-0790f923e4de\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") "
Aug 01 00:20:38.978768 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.978709    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhmfk\" (UniqueName: \"kubernetes.io/projected/ae1c5867-4578-4cf7-bc85-0790f923e4de-kube-api-access-lhmfk\") pod \"ae1c5867-4578-4cf7-bc85-0790f923e4de\" (UID: \"ae1c5867-4578-4cf7-bc85-0790f923e4de\") "
Aug 01 00:20:38.978953 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.978901    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae1c5867-4578-4cf7-bc85-0790f923e4de" (UID: "ae1c5867-4578-4cf7-bc85-0790f923e4de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:20:38.979014 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.978953    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-home" (OuterVolumeSpecName: "home") pod "ae1c5867-4578-4cf7-bc85-0790f923e4de" (UID: "ae1c5867-4578-4cf7-bc85-0790f923e4de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:20:38.979014 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.978969    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae1c5867-4578-4cf7-bc85-0790f923e4de" (UID: "ae1c5867-4578-4cf7-bc85-0790f923e4de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:20:38.979127 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.979047    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:20:38.979127 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.979066    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:20:38.979127 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.979083    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:20:38.980572 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.980549    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-dshm" (OuterVolumeSpecName: "dshm") pod "ae1c5867-4578-4cf7-bc85-0790f923e4de" (UID: "ae1c5867-4578-4cf7-bc85-0790f923e4de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:20:38.980572 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.980562    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1c5867-4578-4cf7-bc85-0790f923e4de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae1c5867-4578-4cf7-bc85-0790f923e4de" (UID: "ae1c5867-4578-4cf7-bc85-0790f923e4de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:20:38.980735 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:38.980719    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1c5867-4578-4cf7-bc85-0790f923e4de-kube-api-access-lhmfk" (OuterVolumeSpecName: "kube-api-access-lhmfk") pod "ae1c5867-4578-4cf7-bc85-0790f923e4de" (UID: "ae1c5867-4578-4cf7-bc85-0790f923e4de"). InnerVolumeSpecName "kube-api-access-lhmfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:20:39.079545 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:39.079507    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae1c5867-4578-4cf7-bc85-0790f923e4de-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:20:39.079545 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:39.079534    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5867-4578-4cf7-bc85-0790f923e4de-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:20:39.079735 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:39.079561    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhmfk\" (UniqueName: \"kubernetes.io/projected/ae1c5867-4578-4cf7-bc85-0790f923e4de-kube-api-access-lhmfk\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:20:39.830133 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:39.830099    2551 generic.go:356] "Generic (PLEG): container finished" podID="ae1c5867-4578-4cf7-bc85-0790f923e4de" containerID="117ff5d8a0bf737692160f4526b784c5345a50604a172f7bd17d84dd0482c686" exitCode=0
Aug 01 00:20:39.830552 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:39.830146    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"
Aug 01 00:20:39.830552 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:39.830187    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w" event={"ID":"ae1c5867-4578-4cf7-bc85-0790f923e4de","Type":"ContainerDied","Data":"117ff5d8a0bf737692160f4526b784c5345a50604a172f7bd17d84dd0482c686"}
Aug 01 00:20:39.830552 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:39.830245    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w" event={"ID":"ae1c5867-4578-4cf7-bc85-0790f923e4de","Type":"ContainerDied","Data":"acf1665fc27166927149eefc3ab2c2d4070612a73193072a74aab0a01091f01a"}
Aug 01 00:20:39.830552 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:39.830274    2551 scope.go:117] "RemoveContainer" containerID="117ff5d8a0bf737692160f4526b784c5345a50604a172f7bd17d84dd0482c686"
Aug 01 00:20:39.839184 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:39.839160    2551 scope.go:117] "RemoveContainer" containerID="117ff5d8a0bf737692160f4526b784c5345a50604a172f7bd17d84dd0482c686"
Aug 01 00:20:39.839425 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:20:39.839405    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"117ff5d8a0bf737692160f4526b784c5345a50604a172f7bd17d84dd0482c686\": container with ID starting with 117ff5d8a0bf737692160f4526b784c5345a50604a172f7bd17d84dd0482c686 not found: ID does not exist" containerID="117ff5d8a0bf737692160f4526b784c5345a50604a172f7bd17d84dd0482c686"
Aug 01 00:20:39.839471 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:39.839432    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"117ff5d8a0bf737692160f4526b784c5345a50604a172f7bd17d84dd0482c686"} err="failed to get container status \"117ff5d8a0bf737692160f4526b784c5345a50604a172f7bd17d84dd0482c686\": rpc error: code = NotFound desc = could not find container \"117ff5d8a0bf737692160f4526b784c5345a50604a172f7bd17d84dd0482c686\": container with ID starting with 117ff5d8a0bf737692160f4526b784c5345a50604a172f7bd17d84dd0482c686 not found: ID does not exist"
Aug 01 00:20:39.850703 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:39.850678    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"]
Aug 01 00:20:39.854152 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:39.854130    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-qrl4w"]
Aug 01 00:20:39.946593 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:39.946565    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae1c5867-4578-4cf7-bc85-0790f923e4de" path="/var/lib/kubelet/pods/ae1c5867-4578-4cf7-bc85-0790f923e4de/volumes"
Aug 01 00:20:50.740139 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:20:50.740107    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"
Aug 01 00:21:00.769135 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:00.769100    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:04.092227 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:04.092180    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:06.480170 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:06.480136    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:06.579939 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:06.579881    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:25.766200 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:25.766168    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:30.309374 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:30.309326    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:30.315239 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:30.315206    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"]
Aug 01 00:21:30.364130 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:30.364070    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:30.740538 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:30.740503    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8" podUID="909344cf-c3c5-4034-93cf-500f101a32e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:21:40.740529 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:40.740491    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8" podUID="909344cf-c3c5-4034-93cf-500f101a32e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:21:45.335218 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:45.335150    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8" podUID="909344cf-c3c5-4034-93cf-500f101a32e8" containerName="main" containerID="cri-o://9a89305733c7afe0790d84183767da3d8327b187fe8df56060ae25c65f694b7a" gracePeriod=45
Aug 01 00:21:45.582324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:45.582297    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"
Aug 01 00:21:45.650029 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:45.649947    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909344cf-c3c5-4034-93cf-500f101a32e8-tls-certs\") pod \"909344cf-c3c5-4034-93cf-500f101a32e8\" (UID: \"909344cf-c3c5-4034-93cf-500f101a32e8\") "
Aug 01 00:21:45.650029 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:45.650014    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twstz\" (UniqueName: \"kubernetes.io/projected/909344cf-c3c5-4034-93cf-500f101a32e8-kube-api-access-twstz\") pod \"909344cf-c3c5-4034-93cf-500f101a32e8\" (UID: \"909344cf-c3c5-4034-93cf-500f101a32e8\") "
Aug 01 00:21:45.652197 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:45.652164    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909344cf-c3c5-4034-93cf-500f101a32e8-kube-api-access-twstz" (OuterVolumeSpecName: "kube-api-access-twstz") pod "909344cf-c3c5-4034-93cf-500f101a32e8" (UID: "909344cf-c3c5-4034-93cf-500f101a32e8"). InnerVolumeSpecName "kube-api-access-twstz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:21:45.652308 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:45.652278    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/909344cf-c3c5-4034-93cf-500f101a32e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "909344cf-c3c5-4034-93cf-500f101a32e8" (UID: "909344cf-c3c5-4034-93cf-500f101a32e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:21:45.750739 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:45.750705    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909344cf-c3c5-4034-93cf-500f101a32e8-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:21:45.750739 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:45.750735    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twstz\" (UniqueName: \"kubernetes.io/projected/909344cf-c3c5-4034-93cf-500f101a32e8-kube-api-access-twstz\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:21:46.120078 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:46.120046    2551 generic.go:356] "Generic (PLEG): container finished" podID="909344cf-c3c5-4034-93cf-500f101a32e8" containerID="9a89305733c7afe0790d84183767da3d8327b187fe8df56060ae25c65f694b7a" exitCode=0
Aug 01 00:21:46.120248 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:46.120092    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"
Aug 01 00:21:46.120248 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:46.120113    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8" event={"ID":"909344cf-c3c5-4034-93cf-500f101a32e8","Type":"ContainerDied","Data":"9a89305733c7afe0790d84183767da3d8327b187fe8df56060ae25c65f694b7a"}
Aug 01 00:21:46.120248 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:46.120168    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8" event={"ID":"909344cf-c3c5-4034-93cf-500f101a32e8","Type":"ContainerDied","Data":"cca1ea913aec685b493878d81b249724e85c68d39ff9ee252103c4c58f0027b1"}
Aug 01 00:21:46.120248 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:46.120193    2551 scope.go:117] "RemoveContainer" containerID="9a89305733c7afe0790d84183767da3d8327b187fe8df56060ae25c65f694b7a"
Aug 01 00:21:46.131337 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:46.131313    2551 scope.go:117] "RemoveContainer" containerID="9a89305733c7afe0790d84183767da3d8327b187fe8df56060ae25c65f694b7a"
Aug 01 00:21:46.131655 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:21:46.131634    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a89305733c7afe0790d84183767da3d8327b187fe8df56060ae25c65f694b7a\": container with ID starting with 9a89305733c7afe0790d84183767da3d8327b187fe8df56060ae25c65f694b7a not found: ID does not exist" containerID="9a89305733c7afe0790d84183767da3d8327b187fe8df56060ae25c65f694b7a"
Aug 01 00:21:46.131711 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:46.131664    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a89305733c7afe0790d84183767da3d8327b187fe8df56060ae25c65f694b7a"} err="failed to get container status \"9a89305733c7afe0790d84183767da3d8327b187fe8df56060ae25c65f694b7a\": rpc error: code = NotFound desc = could not find container \"9a89305733c7afe0790d84183767da3d8327b187fe8df56060ae25c65f694b7a\": container with ID starting with 9a89305733c7afe0790d84183767da3d8327b187fe8df56060ae25c65f694b7a not found: ID does not exist"
Aug 01 00:21:46.134752 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:46.134728    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"]
Aug 01 00:21:46.136843 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:46.136819    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheq55d8"]
Aug 01 00:21:47.946466 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:47.946432    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="909344cf-c3c5-4034-93cf-500f101a32e8" path="/var/lib/kubelet/pods/909344cf-c3c5-4034-93cf-500f101a32e8/volumes"
Aug 01 00:21:49.861224 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.861191    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"]
Aug 01 00:21:49.861724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.861513    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ba5ce4-b0e2-4086-aab8-1078de4fc30e" containerName="main"
Aug 01 00:21:49.861724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.861525    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ba5ce4-b0e2-4086-aab8-1078de4fc30e" containerName="main"
Aug 01 00:21:49.861724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.861540    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="909344cf-c3c5-4034-93cf-500f101a32e8" containerName="main"
Aug 01 00:21:49.861724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.861548    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="909344cf-c3c5-4034-93cf-500f101a32e8" containerName="main"
Aug 01 00:21:49.861724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.861555    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae1c5867-4578-4cf7-bc85-0790f923e4de" containerName="main"
Aug 01 00:21:49.861724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.861560    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae1c5867-4578-4cf7-bc85-0790f923e4de" containerName="main"
Aug 01 00:21:49.861724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.861621    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae1c5867-4578-4cf7-bc85-0790f923e4de" containerName="main"
Aug 01 00:21:49.861724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.861634    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ba5ce4-b0e2-4086-aab8-1078de4fc30e" containerName="main"
Aug 01 00:21:49.861724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.861641    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="909344cf-c3c5-4034-93cf-500f101a32e8" containerName="main"
Aug 01 00:21:49.866065 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.866037    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:49.869405 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.869191    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Aug 01 00:21:49.869405 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.869376    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Aug 01 00:21:49.869630 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.869609    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-4nw7b\""
Aug 01 00:21:49.869894 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.869874    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Aug 01 00:21:49.873735 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.873713    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"]
Aug 01 00:21:49.896825 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.896788    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:49.986811 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.986779    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33952860-9744-4e57-b3b6-34889aec190c-tls-certs\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:49.987032 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.986815    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rxz9\" (UniqueName: \"kubernetes.io/projected/33952860-9744-4e57-b3b6-34889aec190c-kube-api-access-9rxz9\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:49.987032 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.986899    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-dshm\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:49.987032 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.986991    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-model-cache\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:49.987032 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.987018    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-tmp-dir\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:49.987228 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:49.987078    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-home\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.088160 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.088113    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33952860-9744-4e57-b3b6-34889aec190c-tls-certs\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.088160 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.088163    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rxz9\" (UniqueName: \"kubernetes.io/projected/33952860-9744-4e57-b3b6-34889aec190c-kube-api-access-9rxz9\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.088500 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.088212    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-dshm\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.088500 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.088251    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-model-cache\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.088500 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.088275    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-tmp-dir\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.088500 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.088316    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-home\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.088742 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.088723    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-home\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.088800 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.088759    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-model-cache\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.088800 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.088781    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-tmp-dir\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.090566 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.090544    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-dshm\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.090860 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.090839    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33952860-9744-4e57-b3b6-34889aec190c-tls-certs\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.096049 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.096031    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rxz9\" (UniqueName: \"kubernetes.io/projected/33952860-9744-4e57-b3b6-34889aec190c-kube-api-access-9rxz9\") pod \"late-v2-kserve-86766546f-cwsdx\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.185600 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.185474    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:50.318983 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:50.318949    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"]
Aug 01 00:21:50.321489 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:21:50.321461    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33952860_9744_4e57_b3b6_34889aec190c.slice/crio-79c51ef4982b5f64771ba429a0c89b806c31b90007ba99c9250aca3e9a4b621b WatchSource:0}: Error finding container 79c51ef4982b5f64771ba429a0c89b806c31b90007ba99c9250aca3e9a4b621b: Status 404 returned error can't find the container with id 79c51ef4982b5f64771ba429a0c89b806c31b90007ba99c9250aca3e9a4b621b
Aug 01 00:21:51.141747 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:51.141709    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx" event={"ID":"33952860-9744-4e57-b3b6-34889aec190c","Type":"ContainerStarted","Data":"35ed7f18328f343e8ca13cda55a48fbf24269d89bc3bb179299de043b2ea5232"}
Aug 01 00:21:51.142348 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:51.141775    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx" event={"ID":"33952860-9744-4e57-b3b6-34889aec190c","Type":"ContainerStarted","Data":"79c51ef4982b5f64771ba429a0c89b806c31b90007ba99c9250aca3e9a4b621b"}
Aug 01 00:21:51.157565 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:51.157518    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx" podStartSLOduration=2.157504615 podStartE2EDuration="2.157504615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:21:51.156238516 +0000 UTC m=+2153.850285933" watchObservedRunningTime="2026-08-01 00:21:51.157504615 +0000 UTC m=+2153.851552031"
Aug 01 00:21:51.186311 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:51.186283    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:21:51.516645 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:51.516614    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"]
Aug 01 00:21:51.519799 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:51.519765    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:51.554344 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:51.554310    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl" podUID="eef1d222-953c-41c5-9ab6-92e9a4ef4c1f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:21:51.606941 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:21:51.606889    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:00.186264 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:00.186227    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:22:00.198863 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:00.198833    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:22:01.197269 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:01.197239    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:22:01.554248 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:01.554212    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl" podUID="eef1d222-953c-41c5-9ab6-92e9a4ef4c1f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:22:06.536430 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:06.536362    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl" podUID="eef1d222-953c-41c5-9ab6-92e9a4ef4c1f" containerName="main" containerID="cri-o://234393eb1662ea5448c49ea687dc32a210aef6cf940a40cf8724183cfa57906e" gracePeriod=45
Aug 01 00:22:06.787385 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:06.787320    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"
Aug 01 00:22:06.936610 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:06.936574    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eef1d222-953c-41c5-9ab6-92e9a4ef4c1f-tls-certs\") pod \"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f\" (UID: \"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f\") "
Aug 01 00:22:06.936793 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:06.936729    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dppl7\" (UniqueName: \"kubernetes.io/projected/eef1d222-953c-41c5-9ab6-92e9a4ef4c1f-kube-api-access-dppl7\") pod \"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f\" (UID: \"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f\") "
Aug 01 00:22:06.938852 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:06.938820    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef1d222-953c-41c5-9ab6-92e9a4ef4c1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eef1d222-953c-41c5-9ab6-92e9a4ef4c1f" (UID: "eef1d222-953c-41c5-9ab6-92e9a4ef4c1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:22:06.938852 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:06.938841    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef1d222-953c-41c5-9ab6-92e9a4ef4c1f-kube-api-access-dppl7" (OuterVolumeSpecName: "kube-api-access-dppl7") pod "eef1d222-953c-41c5-9ab6-92e9a4ef4c1f" (UID: "eef1d222-953c-41c5-9ab6-92e9a4ef4c1f"). InnerVolumeSpecName "kube-api-access-dppl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:22:07.038004 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:07.037896    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dppl7\" (UniqueName: \"kubernetes.io/projected/eef1d222-953c-41c5-9ab6-92e9a4ef4c1f-kube-api-access-dppl7\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:22:07.038004 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:07.037946    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eef1d222-953c-41c5-9ab6-92e9a4ef4c1f-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:22:07.223808 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:07.223769    2551 generic.go:356] "Generic (PLEG): container finished" podID="eef1d222-953c-41c5-9ab6-92e9a4ef4c1f" containerID="234393eb1662ea5448c49ea687dc32a210aef6cf940a40cf8724183cfa57906e" exitCode=0
Aug 01 00:22:07.224002 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:07.223819    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"
Aug 01 00:22:07.224002 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:07.223837    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl" event={"ID":"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f","Type":"ContainerDied","Data":"234393eb1662ea5448c49ea687dc32a210aef6cf940a40cf8724183cfa57906e"}
Aug 01 00:22:07.224002 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:07.223880    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl" event={"ID":"eef1d222-953c-41c5-9ab6-92e9a4ef4c1f","Type":"ContainerDied","Data":"666594fe77994a90d9f28d94f903571e223ab3fa15eda7760b91e6d2484f44bd"}
Aug 01 00:22:07.224002 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:07.223903    2551 scope.go:117] "RemoveContainer" containerID="234393eb1662ea5448c49ea687dc32a210aef6cf940a40cf8724183cfa57906e"
Aug 01 00:22:07.232896 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:07.232878    2551 scope.go:117] "RemoveContainer" containerID="234393eb1662ea5448c49ea687dc32a210aef6cf940a40cf8724183cfa57906e"
Aug 01 00:22:07.233193 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:22:07.233174    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234393eb1662ea5448c49ea687dc32a210aef6cf940a40cf8724183cfa57906e\": container with ID starting with 234393eb1662ea5448c49ea687dc32a210aef6cf940a40cf8724183cfa57906e not found: ID does not exist" containerID="234393eb1662ea5448c49ea687dc32a210aef6cf940a40cf8724183cfa57906e"
Aug 01 00:22:07.233268 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:07.233200    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234393eb1662ea5448c49ea687dc32a210aef6cf940a40cf8724183cfa57906e"} err="failed to get container status \"234393eb1662ea5448c49ea687dc32a210aef6cf940a40cf8724183cfa57906e\": rpc error: code = NotFound desc = could not find container \"234393eb1662ea5448c49ea687dc32a210aef6cf940a40cf8724183cfa57906e\": container with ID starting with 234393eb1662ea5448c49ea687dc32a210aef6cf940a40cf8724183cfa57906e not found: ID does not exist"
Aug 01 00:22:07.244176 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:07.244152    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"]
Aug 01 00:22:07.247986 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:07.247964    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecmksl"]
Aug 01 00:22:07.946495 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:07.946462    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef1d222-953c-41c5-9ab6-92e9a4ef4c1f" path="/var/lib/kubelet/pods/eef1d222-953c-41c5-9ab6-92e9a4ef4c1f/volumes"
Aug 01 00:22:08.296515 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:08.296482    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:09.894767 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:09.894734    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:13.444190 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:13.444154    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:13.544403 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:13.544366    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:14.600757 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:14.600724    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:19.961612 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:19.961524    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:20.126387 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.126350    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"]
Aug 01 00:22:20.126730 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.126714    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef1d222-953c-41c5-9ab6-92e9a4ef4c1f" containerName="main"
Aug 01 00:22:20.126730 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.126729    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef1d222-953c-41c5-9ab6-92e9a4ef4c1f" containerName="main"
Aug 01 00:22:20.126875 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.126789    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef1d222-953c-41c5-9ab6-92e9a4ef4c1f" containerName="main"
Aug 01 00:22:20.131247 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.131227    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"
Aug 01 00:22:20.133205 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.133184    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Aug 01 00:22:20.133946 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.133897    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Aug 01 00:22:20.133946 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.133906    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Aug 01 00:22:20.134109 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.133980    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-ndrmn\""
Aug 01 00:22:20.134109 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.134003    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-bjjls\""
Aug 01 00:22:20.137072 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.137052    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"]
Aug 01 00:22:20.178205 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.178169    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:20.256789 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.256759    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1620ae-a329-490a-8051-8974ac48e804-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5\" (UID: \"2a1620ae-a329-490a-8051-8974ac48e804\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"
Aug 01 00:22:20.256945 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.256802    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs8bp\" (UniqueName: \"kubernetes.io/projected/2a1620ae-a329-490a-8051-8974ac48e804-kube-api-access-rs8bp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5\" (UID: \"2a1620ae-a329-490a-8051-8974ac48e804\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"
Aug 01 00:22:20.357933 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.357883    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1620ae-a329-490a-8051-8974ac48e804-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5\" (UID: \"2a1620ae-a329-490a-8051-8974ac48e804\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"
Aug 01 00:22:20.358099 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.357943    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs8bp\" (UniqueName: \"kubernetes.io/projected/2a1620ae-a329-490a-8051-8974ac48e804-kube-api-access-rs8bp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5\" (UID: \"2a1620ae-a329-490a-8051-8974ac48e804\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"
Aug 01 00:22:20.360255 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.360231    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1620ae-a329-490a-8051-8974ac48e804-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5\" (UID: \"2a1620ae-a329-490a-8051-8974ac48e804\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"
Aug 01 00:22:20.365097 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.365064    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs8bp\" (UniqueName: \"kubernetes.io/projected/2a1620ae-a329-490a-8051-8974ac48e804-kube-api-access-rs8bp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5\" (UID: \"2a1620ae-a329-490a-8051-8974ac48e804\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"
Aug 01 00:22:20.443146 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.443107    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"
Aug 01 00:22:20.493891 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.493694    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:20.582422 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.582385    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"]
Aug 01 00:22:20.585334 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:22:20.585295    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1620ae_a329_490a_8051_8974ac48e804.slice/crio-09dcb49fbe6706fb0a44870be0b8c1facbd87be8f87de147368ab4ce25ea38a4 WatchSource:0}: Error finding container 09dcb49fbe6706fb0a44870be0b8c1facbd87be8f87de147368ab4ce25ea38a4: Status 404 returned error can't find the container with id 09dcb49fbe6706fb0a44870be0b8c1facbd87be8f87de147368ab4ce25ea38a4
Aug 01 00:22:20.588329 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:20.588303    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:21.284956 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:21.284896    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5" event={"ID":"2a1620ae-a329-490a-8051-8974ac48e804","Type":"ContainerStarted","Data":"a6b8d86a6e444d06af7236c953b6e4b4f6f76767c142ec454e954f19e7d70c74"}
Aug 01 00:22:21.285374 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:21.285009    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5" event={"ID":"2a1620ae-a329-490a-8051-8974ac48e804","Type":"ContainerStarted","Data":"09dcb49fbe6706fb0a44870be0b8c1facbd87be8f87de147368ab4ce25ea38a4"}
Aug 01 00:22:21.285374 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:21.285071    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"
Aug 01 00:22:21.303493 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:21.303424    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5" podStartSLOduration=1.3034032199999999 podStartE2EDuration="1.30340322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:22:21.300851786 +0000 UTC m=+2183.994899205" watchObservedRunningTime="2026-08-01 00:22:21.30340322 +0000 UTC m=+2183.997450637"
Aug 01 00:22:30.100397 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:30.100347    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:30.668672 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:30.668637    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:30.721474 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:30.721442    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:34.379610 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:34.379573    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:34.480123 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:34.480086    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:34.582392 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:34.582358    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:36.265404 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:36.265303    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:39.363098 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:39.363059    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"]
Aug 01 00:22:39.454581 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:39.454530    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:44.140890 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.140854    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"]
Aug 01 00:22:44.144762 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.144735    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.147204 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.147183    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Aug 01 00:22:44.147326 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.147227    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Aug 01 00:22:44.147644 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.147623    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-fkrrt\""
Aug 01 00:22:44.147736 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.147657    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Aug 01 00:22:44.151081 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.151060    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"]
Aug 01 00:22:44.156829 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.156809    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-home\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.156963 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.156856    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-dshm\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.156963 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.156874    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac2c893-f9d0-4553-bf48-d98601138420-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.156963 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.156894    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw7cn\" (UniqueName: \"kubernetes.io/projected/4ac2c893-f9d0-4553-bf48-d98601138420-kube-api-access-gw7cn\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.157137 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.157003    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.157137 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.157065    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.170139 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.170117    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:44.258310 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.258274    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-home\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.258521 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.258340    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-dshm\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.258521 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.258365    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac2c893-f9d0-4553-bf48-d98601138420-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.258521 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.258391    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw7cn\" (UniqueName: \"kubernetes.io/projected/4ac2c893-f9d0-4553-bf48-d98601138420-kube-api-access-gw7cn\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.258521 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.258422    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.258754 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.258739    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.258812 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.258760    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-home\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.258812 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.258776    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.259051 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.259030    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.260832 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.260808    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-dshm\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.260982 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.260942    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac2c893-f9d0-4553-bf48-d98601138420-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.266269 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.266247    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw7cn\" (UniqueName: \"kubernetes.io/projected/4ac2c893-f9d0-4553-bf48-d98601138420-kube-api-access-gw7cn\") pod \"delw-v1-kserve-6cf8cbb847-h52bg\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.457213 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.457112    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:44.589206 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:44.589177    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"]
Aug 01 00:22:44.591654 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:22:44.591624    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac2c893_f9d0_4553_bf48_d98601138420.slice/crio-e7280fd6e9672af0ed6ced42728b5886a9a59ca295421421bad9d618f48e43a4 WatchSource:0}: Error finding container e7280fd6e9672af0ed6ced42728b5886a9a59ca295421421bad9d618f48e43a4: Status 404 returned error can't find the container with id e7280fd6e9672af0ed6ced42728b5886a9a59ca295421421bad9d618f48e43a4
Aug 01 00:22:45.393117 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:45.393077    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg" event={"ID":"4ac2c893-f9d0-4553-bf48-d98601138420","Type":"ContainerStarted","Data":"68082ce32b2662493c0283805d4075928766528b96025c35d4d2140885f31ef5"}
Aug 01 00:22:45.393509 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:45.393133    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg" event={"ID":"4ac2c893-f9d0-4553-bf48-d98601138420","Type":"ContainerStarted","Data":"e7280fd6e9672af0ed6ced42728b5886a9a59ca295421421bad9d618f48e43a4"}
Aug 01 00:22:45.409173 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:45.409122    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg" podStartSLOduration=1.40910672 podStartE2EDuration="1.40910672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:22:45.408134438 +0000 UTC m=+2208.102181856" watchObservedRunningTime="2026-08-01 00:22:45.40910672 +0000 UTC m=+2208.103154136"
Aug 01 00:22:45.457983 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:45.457944    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:52.292026 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:52.291988    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"
Aug 01 00:22:54.381738 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.381688    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx" podUID="33952860-9744-4e57-b3b6-34889aec190c" containerName="main" containerID="cri-o://35ed7f18328f343e8ca13cda55a48fbf24269d89bc3bb179299de043b2ea5232" gracePeriod=45
Aug 01 00:22:54.458085 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.458044    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:54.470540 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.470514    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:54.631862 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.631798    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:22:54.756323 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.756271    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-dshm\") pod \"33952860-9744-4e57-b3b6-34889aec190c\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") "
Aug 01 00:22:54.756508 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.756349    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rxz9\" (UniqueName: \"kubernetes.io/projected/33952860-9744-4e57-b3b6-34889aec190c-kube-api-access-9rxz9\") pod \"33952860-9744-4e57-b3b6-34889aec190c\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") "
Aug 01 00:22:54.756508 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.756390    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33952860-9744-4e57-b3b6-34889aec190c-tls-certs\") pod \"33952860-9744-4e57-b3b6-34889aec190c\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") "
Aug 01 00:22:54.756508 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.756440    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-tmp-dir\") pod \"33952860-9744-4e57-b3b6-34889aec190c\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") "
Aug 01 00:22:54.756508 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.756489    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-home\") pod \"33952860-9744-4e57-b3b6-34889aec190c\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") "
Aug 01 00:22:54.756724 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.756535    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-model-cache\") pod \"33952860-9744-4e57-b3b6-34889aec190c\" (UID: \"33952860-9744-4e57-b3b6-34889aec190c\") "
Aug 01 00:22:54.756779 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.756740    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33952860-9744-4e57-b3b6-34889aec190c" (UID: "33952860-9744-4e57-b3b6-34889aec190c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:22:54.756836 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.756788    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-home" (OuterVolumeSpecName: "home") pod "33952860-9744-4e57-b3b6-34889aec190c" (UID: "33952860-9744-4e57-b3b6-34889aec190c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:22:54.756887 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.756866    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:22:54.756982 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.756887    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-model-cache" (OuterVolumeSpecName: "model-cache") pod "33952860-9744-4e57-b3b6-34889aec190c" (UID: "33952860-9744-4e57-b3b6-34889aec190c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:22:54.758711 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.758679    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33952860-9744-4e57-b3b6-34889aec190c-kube-api-access-9rxz9" (OuterVolumeSpecName: "kube-api-access-9rxz9") pod "33952860-9744-4e57-b3b6-34889aec190c" (UID: "33952860-9744-4e57-b3b6-34889aec190c"). InnerVolumeSpecName "kube-api-access-9rxz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:22:54.758711 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.758699    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-dshm" (OuterVolumeSpecName: "dshm") pod "33952860-9744-4e57-b3b6-34889aec190c" (UID: "33952860-9744-4e57-b3b6-34889aec190c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:22:54.758711 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.758707    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33952860-9744-4e57-b3b6-34889aec190c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33952860-9744-4e57-b3b6-34889aec190c" (UID: "33952860-9744-4e57-b3b6-34889aec190c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:22:54.857517 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.857463    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rxz9\" (UniqueName: \"kubernetes.io/projected/33952860-9744-4e57-b3b6-34889aec190c-kube-api-access-9rxz9\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:22:54.857517 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.857519    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33952860-9744-4e57-b3b6-34889aec190c-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:22:54.857733 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.857536    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:22:54.857733 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.857550    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:22:54.857733 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:54.857563    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33952860-9744-4e57-b3b6-34889aec190c-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:22:55.449223 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:55.449186    2551 generic.go:356] "Generic (PLEG): container finished" podID="33952860-9744-4e57-b3b6-34889aec190c" containerID="35ed7f18328f343e8ca13cda55a48fbf24269d89bc3bb179299de043b2ea5232" exitCode=0
Aug 01 00:22:55.449708 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:55.449242    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"
Aug 01 00:22:55.449708 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:55.449273    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx" event={"ID":"33952860-9744-4e57-b3b6-34889aec190c","Type":"ContainerDied","Data":"35ed7f18328f343e8ca13cda55a48fbf24269d89bc3bb179299de043b2ea5232"}
Aug 01 00:22:55.449708 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:55.449328    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx" event={"ID":"33952860-9744-4e57-b3b6-34889aec190c","Type":"ContainerDied","Data":"79c51ef4982b5f64771ba429a0c89b806c31b90007ba99c9250aca3e9a4b621b"}
Aug 01 00:22:55.449708 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:55.449358    2551 scope.go:117] "RemoveContainer" containerID="35ed7f18328f343e8ca13cda55a48fbf24269d89bc3bb179299de043b2ea5232"
Aug 01 00:22:55.459741 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:55.459708    2551 scope.go:117] "RemoveContainer" containerID="35ed7f18328f343e8ca13cda55a48fbf24269d89bc3bb179299de043b2ea5232"
Aug 01 00:22:55.460078 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:22:55.460049    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ed7f18328f343e8ca13cda55a48fbf24269d89bc3bb179299de043b2ea5232\": container with ID starting with 35ed7f18328f343e8ca13cda55a48fbf24269d89bc3bb179299de043b2ea5232 not found: ID does not exist" containerID="35ed7f18328f343e8ca13cda55a48fbf24269d89bc3bb179299de043b2ea5232"
Aug 01 00:22:55.460209 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:55.460090    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ed7f18328f343e8ca13cda55a48fbf24269d89bc3bb179299de043b2ea5232"} err="failed to get container status \"35ed7f18328f343e8ca13cda55a48fbf24269d89bc3bb179299de043b2ea5232\": rpc error: code = NotFound desc = could not find container \"35ed7f18328f343e8ca13cda55a48fbf24269d89bc3bb179299de043b2ea5232\": container with ID starting with 35ed7f18328f343e8ca13cda55a48fbf24269d89bc3bb179299de043b2ea5232 not found: ID does not exist"
Aug 01 00:22:55.464554 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:55.464532    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:22:55.471553 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:55.471528    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"]
Aug 01 00:22:55.475475 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:55.475454    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-cwsdx"]
Aug 01 00:22:55.947488 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:22:55.947453    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33952860-9744-4e57-b3b6-34889aec190c" path="/var/lib/kubelet/pods/33952860-9744-4e57-b3b6-34889aec190c/volumes"
Aug 01 00:23:16.523369 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:16.523334    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:23:16.660772 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:16.660738    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:23:34.764483 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.764387    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"]
Aug 01 00:23:34.764985 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.764737    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33952860-9744-4e57-b3b6-34889aec190c" containerName="main"
Aug 01 00:23:34.764985 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.764748    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="33952860-9744-4e57-b3b6-34889aec190c" containerName="main"
Aug 01 00:23:34.764985 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.764803    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="33952860-9744-4e57-b3b6-34889aec190c" containerName="main"
Aug 01 00:23:34.767626 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.767607    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.769574 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.769553    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Aug 01 00:23:34.775044 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.775021    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"]
Aug 01 00:23:34.785371 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.785346    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:23:34.820669 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.820640    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.820804 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.820674    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b990bd-c719-4867-b0c6-0de89f062933-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.820804 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.820703    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.820879 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.820819    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-home\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.820879 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.820858    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmzp\" (UniqueName: \"kubernetes.io/projected/45b990bd-c719-4867-b0c6-0de89f062933-kube-api-access-plmzp\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.820993 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.820884    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.922097 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.922062    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-home\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.922285 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.922108    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmzp\" (UniqueName: \"kubernetes.io/projected/45b990bd-c719-4867-b0c6-0de89f062933-kube-api-access-plmzp\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.922285 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.922135    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.922285 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.922177    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.922285 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.922192    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b990bd-c719-4867-b0c6-0de89f062933-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.922285 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.922218    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.922567 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.922547    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-home\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.922622 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.922568    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.922666 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.922650    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.924476 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.924454    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.924770 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.924747    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b990bd-c719-4867-b0c6-0de89f062933-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:34.929425 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:34.929401    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmzp\" (UniqueName: \"kubernetes.io/projected/45b990bd-c719-4867-b0c6-0de89f062933-kube-api-access-plmzp\") pod \"delw-v2-kserve-5b84cc7ddf-p4f55\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:35.078054 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:35.077952    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:35.226377 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:35.226349    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"]
Aug 01 00:23:35.228890 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:23:35.228865    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b990bd_c719_4867_b0c6_0de89f062933.slice/crio-eed2cb55dac9cf1796e510f8849cb6590e6b0292ad2a4e1808876cb8cfc8f82f WatchSource:0}: Error finding container eed2cb55dac9cf1796e510f8849cb6590e6b0292ad2a4e1808876cb8cfc8f82f: Status 404 returned error can't find the container with id eed2cb55dac9cf1796e510f8849cb6590e6b0292ad2a4e1808876cb8cfc8f82f
Aug 01 00:23:35.646570 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:35.646534    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55" event={"ID":"45b990bd-c719-4867-b0c6-0de89f062933","Type":"ContainerStarted","Data":"d67d943da4627408739182e6659a748d956c9072925d29b0ca9952cbf304bb63"}
Aug 01 00:23:35.646742 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:35.646597    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55" event={"ID":"45b990bd-c719-4867-b0c6-0de89f062933","Type":"ContainerStarted","Data":"eed2cb55dac9cf1796e510f8849cb6590e6b0292ad2a4e1808876cb8cfc8f82f"}
Aug 01 00:23:35.663767 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:35.663705    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55" podStartSLOduration=1.663685279 podStartE2EDuration="1.663685279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:23:35.662549589 +0000 UTC m=+2258.356597006" watchObservedRunningTime="2026-08-01 00:23:35.663685279 +0000 UTC m=+2258.357732696"
Aug 01 00:23:36.078934 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:36.078894    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:39.546841 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:39.546799    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:23:39.605712 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:39.605679    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:23:45.078334 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:45.078300    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:45.091978 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:45.091954    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:23:45.702275 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:23:45.702247    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:24:11.046658 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:11.046624    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"]
Aug 01 00:24:11.094707 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:11.094677    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:23.298578 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:23.298537    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"]
Aug 01 00:24:23.310824 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:23.310775    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:23.367279 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:23.367247    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:26.065467 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.065418    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55" podUID="45b990bd-c719-4867-b0c6-0de89f062933" containerName="main" containerID="cri-o://d67d943da4627408739182e6659a748d956c9072925d29b0ca9952cbf304bb63" gracePeriod=45
Aug 01 00:24:26.318098 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.318041    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:24:26.375109 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.375069    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-model-cache\") pod \"45b990bd-c719-4867-b0c6-0de89f062933\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") "
Aug 01 00:24:26.375289 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.375139    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plmzp\" (UniqueName: \"kubernetes.io/projected/45b990bd-c719-4867-b0c6-0de89f062933-kube-api-access-plmzp\") pod \"45b990bd-c719-4867-b0c6-0de89f062933\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") "
Aug 01 00:24:26.375289 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.375167    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-home\") pod \"45b990bd-c719-4867-b0c6-0de89f062933\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") "
Aug 01 00:24:26.375289 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.375212    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b990bd-c719-4867-b0c6-0de89f062933-tls-certs\") pod \"45b990bd-c719-4867-b0c6-0de89f062933\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") "
Aug 01 00:24:26.375289 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.375247    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-tmp-dir\") pod \"45b990bd-c719-4867-b0c6-0de89f062933\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") "
Aug 01 00:24:26.375516 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.375305    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-dshm\") pod \"45b990bd-c719-4867-b0c6-0de89f062933\" (UID: \"45b990bd-c719-4867-b0c6-0de89f062933\") "
Aug 01 00:24:26.375516 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.375355    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-model-cache" (OuterVolumeSpecName: "model-cache") pod "45b990bd-c719-4867-b0c6-0de89f062933" (UID: "45b990bd-c719-4867-b0c6-0de89f062933"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:26.375516 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.375435    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-home" (OuterVolumeSpecName: "home") pod "45b990bd-c719-4867-b0c6-0de89f062933" (UID: "45b990bd-c719-4867-b0c6-0de89f062933"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:26.375696 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.375588    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:26.375696 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.375608    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:26.375696 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.375637    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45b990bd-c719-4867-b0c6-0de89f062933" (UID: "45b990bd-c719-4867-b0c6-0de89f062933"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:26.377591 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.377560    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b990bd-c719-4867-b0c6-0de89f062933-kube-api-access-plmzp" (OuterVolumeSpecName: "kube-api-access-plmzp") pod "45b990bd-c719-4867-b0c6-0de89f062933" (UID: "45b990bd-c719-4867-b0c6-0de89f062933"). InnerVolumeSpecName "kube-api-access-plmzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:24:26.377703 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.377602    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b990bd-c719-4867-b0c6-0de89f062933-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45b990bd-c719-4867-b0c6-0de89f062933" (UID: "45b990bd-c719-4867-b0c6-0de89f062933"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:24:26.377703 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.377635    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-dshm" (OuterVolumeSpecName: "dshm") pod "45b990bd-c719-4867-b0c6-0de89f062933" (UID: "45b990bd-c719-4867-b0c6-0de89f062933"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:26.476909 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.476861    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plmzp\" (UniqueName: \"kubernetes.io/projected/45b990bd-c719-4867-b0c6-0de89f062933-kube-api-access-plmzp\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:26.476909 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.476903    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b990bd-c719-4867-b0c6-0de89f062933-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:26.477142 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.476973    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:26.477142 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.476986    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b990bd-c719-4867-b0c6-0de89f062933-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:26.883889 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.883849    2551 generic.go:356] "Generic (PLEG): container finished" podID="45b990bd-c719-4867-b0c6-0de89f062933" containerID="d67d943da4627408739182e6659a748d956c9072925d29b0ca9952cbf304bb63" exitCode=0
Aug 01 00:24:26.884114 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.883904    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"
Aug 01 00:24:26.884114 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.883961    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55" event={"ID":"45b990bd-c719-4867-b0c6-0de89f062933","Type":"ContainerDied","Data":"d67d943da4627408739182e6659a748d956c9072925d29b0ca9952cbf304bb63"}
Aug 01 00:24:26.884114 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.884031    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55" event={"ID":"45b990bd-c719-4867-b0c6-0de89f062933","Type":"ContainerDied","Data":"eed2cb55dac9cf1796e510f8849cb6590e6b0292ad2a4e1808876cb8cfc8f82f"}
Aug 01 00:24:26.884114 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.884064    2551 scope.go:117] "RemoveContainer" containerID="d67d943da4627408739182e6659a748d956c9072925d29b0ca9952cbf304bb63"
Aug 01 00:24:26.892747 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.892719    2551 scope.go:117] "RemoveContainer" containerID="d67d943da4627408739182e6659a748d956c9072925d29b0ca9952cbf304bb63"
Aug 01 00:24:26.893017 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:24:26.892997    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67d943da4627408739182e6659a748d956c9072925d29b0ca9952cbf304bb63\": container with ID starting with d67d943da4627408739182e6659a748d956c9072925d29b0ca9952cbf304bb63 not found: ID does not exist" containerID="d67d943da4627408739182e6659a748d956c9072925d29b0ca9952cbf304bb63"
Aug 01 00:24:26.893076 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.893025    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67d943da4627408739182e6659a748d956c9072925d29b0ca9952cbf304bb63"} err="failed to get container status \"d67d943da4627408739182e6659a748d956c9072925d29b0ca9952cbf304bb63\": rpc error: code = NotFound desc = could not find container \"d67d943da4627408739182e6659a748d956c9072925d29b0ca9952cbf304bb63\": container with ID starting with d67d943da4627408739182e6659a748d956c9072925d29b0ca9952cbf304bb63 not found: ID does not exist"
Aug 01 00:24:26.904098 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.904067    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"]
Aug 01 00:24:26.909132 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.909114    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-p4f55"]
Aug 01 00:24:26.976308 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:26.976270    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:27.072440 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:27.072405    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"]
Aug 01 00:24:27.075521 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:27.075495    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:27.153589 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:27.153502    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:27.951684 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:27.951648    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b990bd-c719-4867-b0c6-0de89f062933" path="/var/lib/kubelet/pods/45b990bd-c719-4867-b0c6-0de89f062933/volumes"
Aug 01 00:24:32.050782 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:32.050746    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:32.291736 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:32.291697    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5" podUID="2a1620ae-a329-490a-8051-8974ac48e804" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:24:35.590273 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:35.590239    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:38.321626 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.321583    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5" podUID="2a1620ae-a329-490a-8051-8974ac48e804" containerName="main" containerID="cri-o://a6b8d86a6e444d06af7236c953b6e4b4f6f76767c142ec454e954f19e7d70c74" gracePeriod=45
Aug 01 00:24:38.571336 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.571305    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"
Aug 01 00:24:38.675262 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.675154    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs8bp\" (UniqueName: \"kubernetes.io/projected/2a1620ae-a329-490a-8051-8974ac48e804-kube-api-access-rs8bp\") pod \"2a1620ae-a329-490a-8051-8974ac48e804\" (UID: \"2a1620ae-a329-490a-8051-8974ac48e804\") "
Aug 01 00:24:38.675262 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.675230    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1620ae-a329-490a-8051-8974ac48e804-tls-certs\") pod \"2a1620ae-a329-490a-8051-8974ac48e804\" (UID: \"2a1620ae-a329-490a-8051-8974ac48e804\") "
Aug 01 00:24:38.677496 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.677467    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1620ae-a329-490a-8051-8974ac48e804-kube-api-access-rs8bp" (OuterVolumeSpecName: "kube-api-access-rs8bp") pod "2a1620ae-a329-490a-8051-8974ac48e804" (UID: "2a1620ae-a329-490a-8051-8974ac48e804"). InnerVolumeSpecName "kube-api-access-rs8bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:24:38.677567 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.677496    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1620ae-a329-490a-8051-8974ac48e804-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a1620ae-a329-490a-8051-8974ac48e804" (UID: "2a1620ae-a329-490a-8051-8974ac48e804"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:24:38.776501 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.776464    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs8bp\" (UniqueName: \"kubernetes.io/projected/2a1620ae-a329-490a-8051-8974ac48e804-kube-api-access-rs8bp\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:38.776501 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.776494    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1620ae-a329-490a-8051-8974ac48e804-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:38.939312 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.939223    2551 generic.go:356] "Generic (PLEG): container finished" podID="2a1620ae-a329-490a-8051-8974ac48e804" containerID="a6b8d86a6e444d06af7236c953b6e4b4f6f76767c142ec454e954f19e7d70c74" exitCode=0
Aug 01 00:24:38.939312 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.939270    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"
Aug 01 00:24:38.939541 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.939312    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5" event={"ID":"2a1620ae-a329-490a-8051-8974ac48e804","Type":"ContainerDied","Data":"a6b8d86a6e444d06af7236c953b6e4b4f6f76767c142ec454e954f19e7d70c74"}
Aug 01 00:24:38.939541 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.939371    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5" event={"ID":"2a1620ae-a329-490a-8051-8974ac48e804","Type":"ContainerDied","Data":"09dcb49fbe6706fb0a44870be0b8c1facbd87be8f87de147368ab4ce25ea38a4"}
Aug 01 00:24:38.939541 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.939403    2551 scope.go:117] "RemoveContainer" containerID="a6b8d86a6e444d06af7236c953b6e4b4f6f76767c142ec454e954f19e7d70c74"
Aug 01 00:24:38.947731 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.947713    2551 scope.go:117] "RemoveContainer" containerID="a6b8d86a6e444d06af7236c953b6e4b4f6f76767c142ec454e954f19e7d70c74"
Aug 01 00:24:38.948007 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:24:38.947985    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b8d86a6e444d06af7236c953b6e4b4f6f76767c142ec454e954f19e7d70c74\": container with ID starting with a6b8d86a6e444d06af7236c953b6e4b4f6f76767c142ec454e954f19e7d70c74 not found: ID does not exist" containerID="a6b8d86a6e444d06af7236c953b6e4b4f6f76767c142ec454e954f19e7d70c74"
Aug 01 00:24:38.948108 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.948014    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b8d86a6e444d06af7236c953b6e4b4f6f76767c142ec454e954f19e7d70c74"} err="failed to get container status \"a6b8d86a6e444d06af7236c953b6e4b4f6f76767c142ec454e954f19e7d70c74\": rpc error: code = NotFound desc = could not find container \"a6b8d86a6e444d06af7236c953b6e4b4f6f76767c142ec454e954f19e7d70c74\": container with ID starting with a6b8d86a6e444d06af7236c953b6e4b4f6f76767c142ec454e954f19e7d70c74 not found: ID does not exist"
Aug 01 00:24:38.958395 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.958374    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"]
Aug 01 00:24:38.962175 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:38.962157    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schefdmr5"]
Aug 01 00:24:39.947127 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:39.947097    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a1620ae-a329-490a-8051-8974ac48e804" path="/var/lib/kubelet/pods/2a1620ae-a329-490a-8051-8974ac48e804/volumes"
Aug 01 00:24:42.090761 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.090714    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg" podUID="4ac2c893-f9d0-4553-bf48-d98601138420" containerName="main" containerID="cri-o://68082ce32b2662493c0283805d4075928766528b96025c35d4d2140885f31ef5" gracePeriod=45
Aug 01 00:24:42.339613 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.339588    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:24:42.404534 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.404451    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-home\") pod \"4ac2c893-f9d0-4553-bf48-d98601138420\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") "
Aug 01 00:24:42.404534 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.404500    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac2c893-f9d0-4553-bf48-d98601138420-tls-certs\") pod \"4ac2c893-f9d0-4553-bf48-d98601138420\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") "
Aug 01 00:24:42.404758 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.404551    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-dshm\") pod \"4ac2c893-f9d0-4553-bf48-d98601138420\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") "
Aug 01 00:24:42.404758 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.404566    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-tmp-dir\") pod \"4ac2c893-f9d0-4553-bf48-d98601138420\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") "
Aug 01 00:24:42.404758 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.404592    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw7cn\" (UniqueName: \"kubernetes.io/projected/4ac2c893-f9d0-4553-bf48-d98601138420-kube-api-access-gw7cn\") pod \"4ac2c893-f9d0-4553-bf48-d98601138420\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") "
Aug 01 00:24:42.404758 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.404650    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-model-cache\") pod \"4ac2c893-f9d0-4553-bf48-d98601138420\" (UID: \"4ac2c893-f9d0-4553-bf48-d98601138420\") "
Aug 01 00:24:42.404758 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.404724    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-home" (OuterVolumeSpecName: "home") pod "4ac2c893-f9d0-4553-bf48-d98601138420" (UID: "4ac2c893-f9d0-4553-bf48-d98601138420"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:42.405070 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.404861    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ac2c893-f9d0-4553-bf48-d98601138420" (UID: "4ac2c893-f9d0-4553-bf48-d98601138420"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:42.405070 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.404884    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:42.405070 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.404960    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ac2c893-f9d0-4553-bf48-d98601138420" (UID: "4ac2c893-f9d0-4553-bf48-d98601138420"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:42.406769 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.406743    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-dshm" (OuterVolumeSpecName: "dshm") pod "4ac2c893-f9d0-4553-bf48-d98601138420" (UID: "4ac2c893-f9d0-4553-bf48-d98601138420"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:42.406877 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.406813    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac2c893-f9d0-4553-bf48-d98601138420-kube-api-access-gw7cn" (OuterVolumeSpecName: "kube-api-access-gw7cn") pod "4ac2c893-f9d0-4553-bf48-d98601138420" (UID: "4ac2c893-f9d0-4553-bf48-d98601138420"). InnerVolumeSpecName "kube-api-access-gw7cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:24:42.406943 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.406879    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac2c893-f9d0-4553-bf48-d98601138420-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ac2c893-f9d0-4553-bf48-d98601138420" (UID: "4ac2c893-f9d0-4553-bf48-d98601138420"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:24:42.506235 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.506184    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:42.506235 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.506226    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:42.506235 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.506239    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw7cn\" (UniqueName: \"kubernetes.io/projected/4ac2c893-f9d0-4553-bf48-d98601138420-kube-api-access-gw7cn\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:42.506235 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.506249    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ac2c893-f9d0-4553-bf48-d98601138420-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:42.506235 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.506258    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ac2c893-f9d0-4553-bf48-d98601138420-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:24:42.959258 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.959220    2551 generic.go:356] "Generic (PLEG): container finished" podID="4ac2c893-f9d0-4553-bf48-d98601138420" containerID="68082ce32b2662493c0283805d4075928766528b96025c35d4d2140885f31ef5" exitCode=0
Aug 01 00:24:42.959428 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.959269    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"
Aug 01 00:24:42.959428 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.959305    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg" event={"ID":"4ac2c893-f9d0-4553-bf48-d98601138420","Type":"ContainerDied","Data":"68082ce32b2662493c0283805d4075928766528b96025c35d4d2140885f31ef5"}
Aug 01 00:24:42.959428 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.959360    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg" event={"ID":"4ac2c893-f9d0-4553-bf48-d98601138420","Type":"ContainerDied","Data":"e7280fd6e9672af0ed6ced42728b5886a9a59ca295421421bad9d618f48e43a4"}
Aug 01 00:24:42.959428 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.959384    2551 scope.go:117] "RemoveContainer" containerID="68082ce32b2662493c0283805d4075928766528b96025c35d4d2140885f31ef5"
Aug 01 00:24:42.967810 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.967793    2551 scope.go:117] "RemoveContainer" containerID="68082ce32b2662493c0283805d4075928766528b96025c35d4d2140885f31ef5"
Aug 01 00:24:42.968067 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:24:42.968045    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68082ce32b2662493c0283805d4075928766528b96025c35d4d2140885f31ef5\": container with ID starting with 68082ce32b2662493c0283805d4075928766528b96025c35d4d2140885f31ef5 not found: ID does not exist" containerID="68082ce32b2662493c0283805d4075928766528b96025c35d4d2140885f31ef5"
Aug 01 00:24:42.968118 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.968076    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68082ce32b2662493c0283805d4075928766528b96025c35d4d2140885f31ef5"} err="failed to get container status \"68082ce32b2662493c0283805d4075928766528b96025c35d4d2140885f31ef5\": rpc error: code = NotFound desc = could not find container \"68082ce32b2662493c0283805d4075928766528b96025c35d4d2140885f31ef5\": container with ID starting with 68082ce32b2662493c0283805d4075928766528b96025c35d4d2140885f31ef5 not found: ID does not exist"
Aug 01 00:24:42.979660 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.979637    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"]
Aug 01 00:24:42.982748 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:42.982728    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-h52bg"]
Aug 01 00:24:43.324317 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.324283    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:43.364319 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.364285    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"]
Aug 01 00:24:43.364589 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.364577    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac2c893-f9d0-4553-bf48-d98601138420" containerName="main"
Aug 01 00:24:43.364635 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.364590    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac2c893-f9d0-4553-bf48-d98601138420" containerName="main"
Aug 01 00:24:43.364635 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.364603    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1620ae-a329-490a-8051-8974ac48e804" containerName="main"
Aug 01 00:24:43.364635 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.364609    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1620ae-a329-490a-8051-8974ac48e804" containerName="main"
Aug 01 00:24:43.364635 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.364619    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b990bd-c719-4867-b0c6-0de89f062933" containerName="main"
Aug 01 00:24:43.364635 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.364625    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b990bd-c719-4867-b0c6-0de89f062933" containerName="main"
Aug 01 00:24:43.364804 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.364669    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b990bd-c719-4867-b0c6-0de89f062933" containerName="main"
Aug 01 00:24:43.364804 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.364678    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ac2c893-f9d0-4553-bf48-d98601138420" containerName="main"
Aug 01 00:24:43.364804 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.364686    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1620ae-a329-490a-8051-8974ac48e804" containerName="main"
Aug 01 00:24:43.368797 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.368780    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.371212 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.371193    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Aug 01 00:24:43.371295 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.371219    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Aug 01 00:24:43.372027 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.371988    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-w467j\""
Aug 01 00:24:43.372027 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.371998    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Aug 01 00:24:43.376874 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.376853    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"]
Aug 01 00:24:43.415704 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.415657    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/268ebfec-a51e-4021-b11f-501c2c0e420c-home\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.415869 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.415708    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/268ebfec-a51e-4021-b11f-501c2c0e420c-dshm\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.415869 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.415759    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/268ebfec-a51e-4021-b11f-501c2c0e420c-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.415869 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.415782    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268ebfec-a51e-4021-b11f-501c2c0e420c-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.416066 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.415873    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp5bn\" (UniqueName: \"kubernetes.io/projected/268ebfec-a51e-4021-b11f-501c2c0e420c-kube-api-access-wp5bn\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.416066 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.416002    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/268ebfec-a51e-4021-b11f-501c2c0e420c-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.421506 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.421485    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:43.517221 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.517191    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/268ebfec-a51e-4021-b11f-501c2c0e420c-home\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.517221 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.517219    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/268ebfec-a51e-4021-b11f-501c2c0e420c-dshm\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.517469 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.517236    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/268ebfec-a51e-4021-b11f-501c2c0e420c-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.517469 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.517252    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268ebfec-a51e-4021-b11f-501c2c0e420c-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.517469 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.517299    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp5bn\" (UniqueName: \"kubernetes.io/projected/268ebfec-a51e-4021-b11f-501c2c0e420c-kube-api-access-wp5bn\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.517469 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.517368    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/268ebfec-a51e-4021-b11f-501c2c0e420c-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.517633 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.517611    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/268ebfec-a51e-4021-b11f-501c2c0e420c-home\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.517678 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.517655    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/268ebfec-a51e-4021-b11f-501c2c0e420c-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.517725 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.517692    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268ebfec-a51e-4021-b11f-501c2c0e420c-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.519588 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.519569    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/268ebfec-a51e-4021-b11f-501c2c0e420c-dshm\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.519826 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.519810    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/268ebfec-a51e-4021-b11f-501c2c0e420c-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.531383 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.531361    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp5bn\" (UniqueName: \"kubernetes.io/projected/268ebfec-a51e-4021-b11f-501c2c0e420c-kube-api-access-wp5bn\") pod \"roll-v2-kserve-d4dbf9d67-2hs7r\" (UID: \"268ebfec-a51e-4021-b11f-501c2c0e420c\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.538606 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.538583    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:43.679636 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.679550    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:43.805670 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.805640    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"]
Aug 01 00:24:43.807774 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:24:43.807744    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268ebfec_a51e_4021_b11f_501c2c0e420c.slice/crio-edee210d19b2d565ea17de134f0c8d92a52de2e2fc5b4c7f8a56521dea256946 WatchSource:0}: Error finding container edee210d19b2d565ea17de134f0c8d92a52de2e2fc5b4c7f8a56521dea256946: Status 404 returned error can't find the container with id edee210d19b2d565ea17de134f0c8d92a52de2e2fc5b4c7f8a56521dea256946
Aug 01 00:24:43.948755 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.948670    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ac2c893-f9d0-4553-bf48-d98601138420" path="/var/lib/kubelet/pods/4ac2c893-f9d0-4553-bf48-d98601138420/volumes"
Aug 01 00:24:43.965137 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.965095    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r" event={"ID":"268ebfec-a51e-4021-b11f-501c2c0e420c","Type":"ContainerStarted","Data":"5b794876fe6319b98e995ffaf7508f9083d914557b452c9d0f815552e324ac24"}
Aug 01 00:24:43.965311 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.965158    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r" event={"ID":"268ebfec-a51e-4021-b11f-501c2c0e420c","Type":"ContainerStarted","Data":"edee210d19b2d565ea17de134f0c8d92a52de2e2fc5b4c7f8a56521dea256946"}
Aug 01 00:24:43.980511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:43.980463    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r" podStartSLOduration=0.980449662 podStartE2EDuration="980.449662ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:24:43.979347573 +0000 UTC m=+2326.673394990" watchObservedRunningTime="2026-08-01 00:24:43.980449662 +0000 UTC m=+2326.674497106"
Aug 01 00:24:44.680676 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:44.680628    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:49.147223 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:49.147183    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:53.679975 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:53.679889    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:53.692414 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:53.692386    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:54.019208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:54.019173    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-2hs7r"
Aug 01 00:24:54.659040 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:54.659001    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:54.759688 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:24:54.759645    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:08.803182 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:08.803145    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:09.649379 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:09.649341    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:14.587437 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:14.587390    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:14.707680 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:14.707646    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:26.971808 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:26.971769    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:27.198144 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.198105    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"]
Aug 01 00:25:27.208502 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.208476    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"
Aug 01 00:25:27.210900 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.210866    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"]
Aug 01 00:25:27.211822 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.211797    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-2trwt\""
Aug 01 00:25:27.212010 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.211988    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Aug 01 00:25:27.212158 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.212131    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-vvf2c\""
Aug 01 00:25:27.212226 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.211912    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Aug 01 00:25:27.212285 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.211796    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Aug 01 00:25:27.233345 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.233266    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:27.311282 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.311237    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrms\" (UniqueName: \"kubernetes.io/projected/f5e30159-1db1-4685-96b3-c363bf43e23a-kube-api-access-5lrms\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7\" (UID: \"f5e30159-1db1-4685-96b3-c363bf43e23a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"
Aug 01 00:25:27.311282 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.311281    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5e30159-1db1-4685-96b3-c363bf43e23a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7\" (UID: \"f5e30159-1db1-4685-96b3-c363bf43e23a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"
Aug 01 00:25:27.412286 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.412248    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrms\" (UniqueName: \"kubernetes.io/projected/f5e30159-1db1-4685-96b3-c363bf43e23a-kube-api-access-5lrms\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7\" (UID: \"f5e30159-1db1-4685-96b3-c363bf43e23a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"
Aug 01 00:25:27.412459 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.412303    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5e30159-1db1-4685-96b3-c363bf43e23a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7\" (UID: \"f5e30159-1db1-4685-96b3-c363bf43e23a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"
Aug 01 00:25:27.414801 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.414775    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5e30159-1db1-4685-96b3-c363bf43e23a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7\" (UID: \"f5e30159-1db1-4685-96b3-c363bf43e23a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"
Aug 01 00:25:27.419562 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.419540    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrms\" (UniqueName: \"kubernetes.io/projected/f5e30159-1db1-4685-96b3-c363bf43e23a-kube-api-access-5lrms\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7\" (UID: \"f5e30159-1db1-4685-96b3-c363bf43e23a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"
Aug 01 00:25:27.522769 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.522668    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"
Aug 01 00:25:27.650866 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:27.650833    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"]
Aug 01 00:25:27.653817 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:25:27.653791    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e30159_1db1_4685_96b3_c363bf43e23a.slice/crio-9ebc941ee788e550ef7147703a857969bd35554b87672f70c6e37ac40aaa8c0e WatchSource:0}: Error finding container 9ebc941ee788e550ef7147703a857969bd35554b87672f70c6e37ac40aaa8c0e: Status 404 returned error can't find the container with id 9ebc941ee788e550ef7147703a857969bd35554b87672f70c6e37ac40aaa8c0e
Aug 01 00:25:28.151427 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:28.151386    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7" event={"ID":"f5e30159-1db1-4685-96b3-c363bf43e23a","Type":"ContainerStarted","Data":"1c03d0bd23716ac323f27de02a38f6945ce08cd004f1d6ecda9d79b4e62f025f"}
Aug 01 00:25:28.151832 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:28.151454    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7" event={"ID":"f5e30159-1db1-4685-96b3-c363bf43e23a","Type":"ContainerStarted","Data":"9ebc941ee788e550ef7147703a857969bd35554b87672f70c6e37ac40aaa8c0e"}
Aug 01 00:25:28.151832 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:28.151483    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"
Aug 01 00:25:28.170377 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:28.170314    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7" podStartSLOduration=1.170292719 podStartE2EDuration="1.170292719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:25:28.166905634 +0000 UTC m=+2370.860953050" watchObservedRunningTime="2026-08-01 00:25:28.170292719 +0000 UTC m=+2370.864340138"
Aug 01 00:25:38.341736 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:38.341692    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:39.346601 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:39.346562    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:45.503102 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:45.503065    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:49.310864 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:49.310836    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-rollback-b8fdb441_roll-v2-kserve-d4dbf9d67-2hs7r_268ebfec-a51e-4021-b11f-501c2c0e420c/main/0.log"
Aug 01 00:25:50.372826 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:50.372786    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:50.469040 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:50.468994    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:59.157384 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:25:59.157352    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"
Aug 01 00:26:01.191010 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:26:01.190969    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:26:06.284839 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:26:06.284809    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:26:40.974193 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:26:40.974151    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:03.529008 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.528962    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:03.710027 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.709989    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:03.751670 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.751639    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"]
Aug 01 00:27:03.755301 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.755130    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.758270 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.758240    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Aug 01 00:27:03.758439 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.758363    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-v5hsk\""
Aug 01 00:27:03.758439 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.758388    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Aug 01 00:27:03.758439 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.758254    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Aug 01 00:27:03.773260 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.773231    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"]
Aug 01 00:27:03.797568 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.797453    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:03.858974 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.858940    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:03.860011 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.859992    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-podinfo\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.860085 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.860021    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2c0880-fd5c-4872-bacc-652bfdd61b39-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.860085 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.860042    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-workload-certs\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.860163 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.860099    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-envoy\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.860198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.860161    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-token\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.860198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.860180    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dn4j\" (UniqueName: \"kubernetes.io/projected/be2c0880-fd5c-4872-bacc-652bfdd61b39-kube-api-access-6dn4j\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.860263 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.860209    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-workload-socket\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.860263 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.860225    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-credential-socket\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.860263 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.860246    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-data\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.960798 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.960750    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-token\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.961042 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.960812    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dn4j\" (UniqueName: \"kubernetes.io/projected/be2c0880-fd5c-4872-bacc-652bfdd61b39-kube-api-access-6dn4j\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.961042 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.960872    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-workload-socket\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.961198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.961036    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-credential-socket\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.961198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.961090    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-data\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.961198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.961148    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-podinfo\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.961198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.961179    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2c0880-fd5c-4872-bacc-652bfdd61b39-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.961430 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.961296    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-workload-certs\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.961430 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.961320    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-workload-socket\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.961430 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.961344    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-envoy\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.961430 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.961406    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-credential-socket\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.961621 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.961510    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-data\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.961621 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.961593    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-workload-certs\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.961858 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.961838    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2c0880-fd5c-4872-bacc-652bfdd61b39-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.963616 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.963590    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-envoy\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.963718 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.963690    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-podinfo\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.968427 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.968387    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-token\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:03.968559 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:03.968519    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dn4j\" (UniqueName: \"kubernetes.io/projected/be2c0880-fd5c-4872-bacc-652bfdd61b39-kube-api-access-6dn4j\") pod \"router-gateway-1-openshift-default-65765d8947-jhv6t\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:04.066323 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:04.066225    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:04.227386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:04.227358    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"]
Aug 01 00:27:04.231236 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:27:04.231201    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2c0880_fd5c_4872_bacc_652bfdd61b39.slice/crio-bb5af7f5226b672466ef64795009abeb3a01688ae69f98e219d9a122b3675247 WatchSource:0}: Error finding container bb5af7f5226b672466ef64795009abeb3a01688ae69f98e219d9a122b3675247: Status 404 returned error can't find the container with id bb5af7f5226b672466ef64795009abeb3a01688ae69f98e219d9a122b3675247
Aug 01 00:27:04.234030 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:04.233992    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:27:04.234144 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:04.234079    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:27:04.234208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:04.234146    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:27:04.574000 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:04.573958    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t" event={"ID":"be2c0880-fd5c-4872-bacc-652bfdd61b39","Type":"ContainerStarted","Data":"0fa8a9ecbe6e0b0963979db201bf0e36c5ed764865fe99223fb45a89a01a1a54"}
Aug 01 00:27:04.574410 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:04.574026    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t" event={"ID":"be2c0880-fd5c-4872-bacc-652bfdd61b39","Type":"ContainerStarted","Data":"bb5af7f5226b672466ef64795009abeb3a01688ae69f98e219d9a122b3675247"}
Aug 01 00:27:05.067233 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:05.067183    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:05.073162 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:05.073135    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:05.090606 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:05.090552    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t" podStartSLOduration=2.0905357589999998 podStartE2EDuration="2.090535759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:27:04.591802634 +0000 UTC m=+2467.285850050" watchObservedRunningTime="2026-08-01 00:27:05.090535759 +0000 UTC m=+2467.784583175"
Aug 01 00:27:05.579310 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:05.579273    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:05.580367 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:05.580346    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:07.285985 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:07.285952    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:07.506288 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:07.506255    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:08.605432 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:08.605384    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:13.960818 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:13.960777    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:25.679856 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:25.679819    2551 generic.go:356] "Generic (PLEG): container finished" podID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerID="0fa8a9ecbe6e0b0963979db201bf0e36c5ed764865fe99223fb45a89a01a1a54" exitCode=0
Aug 01 00:27:25.680429 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:25.679902    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t" event={"ID":"be2c0880-fd5c-4872-bacc-652bfdd61b39","Type":"ContainerDied","Data":"0fa8a9ecbe6e0b0963979db201bf0e36c5ed764865fe99223fb45a89a01a1a54"}
Aug 01 00:27:25.680516 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:25.680460    2551 scope.go:117] "RemoveContainer" containerID="0fa8a9ecbe6e0b0963979db201bf0e36c5ed764865fe99223fb45a89a01a1a54"
Aug 01 00:27:25.681470 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:25.681442    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:27:25.681572 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:25.681523    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:27:25.681633 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:25.681570    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:27:26.687316 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:26.687278    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t" event={"ID":"be2c0880-fd5c-4872-bacc-652bfdd61b39","Type":"ContainerStarted","Data":"33e4190d16ff75b5f59f59e38cc7514431cd21c4aa64fed9c23409d816a4da10"}
Aug 01 00:27:27.066577 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:27.066532    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:27.066809 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:27.066782    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t" podUID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Aug 01 00:27:28.067085 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:28.067041    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t" podUID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Aug 01 00:27:29.067481 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:29.067437    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t" podUID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Aug 01 00:27:30.066797 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:30.066753    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t" podUID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Aug 01 00:27:31.067473 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:31.067423    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t" podUID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Aug 01 00:27:32.140523 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:32.140487    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:32.141053 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:32.140790    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:32.141796 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:32.141775    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:27:45.662615 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:45.662578    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:50.852700 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:50.852537    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:50.853207 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:50.853183    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"]
Aug 01 00:27:50.916812 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:50.916762    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:51.423073 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:51.423037    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:51.522754 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:51.522721    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:59.157269 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:27:59.157228    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7" podUID="f5e30159-1db1-4685-96b3-c363bf43e23a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:28:05.873456 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:05.873404    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7" podUID="f5e30159-1db1-4685-96b3-c363bf43e23a" containerName="main" containerID="cri-o://1c03d0bd23716ac323f27de02a38f6945ce08cd004f1d6ecda9d79b4e62f025f" gracePeriod=45
Aug 01 00:28:06.110808 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.110782    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"
Aug 01 00:28:06.247968 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.247912    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5e30159-1db1-4685-96b3-c363bf43e23a-tls-certs\") pod \"f5e30159-1db1-4685-96b3-c363bf43e23a\" (UID: \"f5e30159-1db1-4685-96b3-c363bf43e23a\") "
Aug 01 00:28:06.248134 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.248036    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lrms\" (UniqueName: \"kubernetes.io/projected/f5e30159-1db1-4685-96b3-c363bf43e23a-kube-api-access-5lrms\") pod \"f5e30159-1db1-4685-96b3-c363bf43e23a\" (UID: \"f5e30159-1db1-4685-96b3-c363bf43e23a\") "
Aug 01 00:28:06.250193 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.250157    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e30159-1db1-4685-96b3-c363bf43e23a-kube-api-access-5lrms" (OuterVolumeSpecName: "kube-api-access-5lrms") pod "f5e30159-1db1-4685-96b3-c363bf43e23a" (UID: "f5e30159-1db1-4685-96b3-c363bf43e23a"). InnerVolumeSpecName "kube-api-access-5lrms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:28:06.250193 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.250175    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5e30159-1db1-4685-96b3-c363bf43e23a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5e30159-1db1-4685-96b3-c363bf43e23a" (UID: "f5e30159-1db1-4685-96b3-c363bf43e23a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:28:06.349394 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.349359    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5e30159-1db1-4685-96b3-c363bf43e23a-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:28:06.349394 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.349386    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lrms\" (UniqueName: \"kubernetes.io/projected/f5e30159-1db1-4685-96b3-c363bf43e23a-kube-api-access-5lrms\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:28:06.869154 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.869118    2551 generic.go:356] "Generic (PLEG): container finished" podID="f5e30159-1db1-4685-96b3-c363bf43e23a" containerID="1c03d0bd23716ac323f27de02a38f6945ce08cd004f1d6ecda9d79b4e62f025f" exitCode=0
Aug 01 00:28:06.869375 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.869164    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"
Aug 01 00:28:06.869375 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.869175    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7" event={"ID":"f5e30159-1db1-4685-96b3-c363bf43e23a","Type":"ContainerDied","Data":"1c03d0bd23716ac323f27de02a38f6945ce08cd004f1d6ecda9d79b4e62f025f"}
Aug 01 00:28:06.869375 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.869229    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7" event={"ID":"f5e30159-1db1-4685-96b3-c363bf43e23a","Type":"ContainerDied","Data":"9ebc941ee788e550ef7147703a857969bd35554b87672f70c6e37ac40aaa8c0e"}
Aug 01 00:28:06.869375 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.869259    2551 scope.go:117] "RemoveContainer" containerID="1c03d0bd23716ac323f27de02a38f6945ce08cd004f1d6ecda9d79b4e62f025f"
Aug 01 00:28:06.878975 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.878835    2551 scope.go:117] "RemoveContainer" containerID="1c03d0bd23716ac323f27de02a38f6945ce08cd004f1d6ecda9d79b4e62f025f"
Aug 01 00:28:06.879192 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:28:06.879125    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c03d0bd23716ac323f27de02a38f6945ce08cd004f1d6ecda9d79b4e62f025f\": container with ID starting with 1c03d0bd23716ac323f27de02a38f6945ce08cd004f1d6ecda9d79b4e62f025f not found: ID does not exist" containerID="1c03d0bd23716ac323f27de02a38f6945ce08cd004f1d6ecda9d79b4e62f025f"
Aug 01 00:28:06.879192 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.879154    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c03d0bd23716ac323f27de02a38f6945ce08cd004f1d6ecda9d79b4e62f025f"} err="failed to get container status \"1c03d0bd23716ac323f27de02a38f6945ce08cd004f1d6ecda9d79b4e62f025f\": rpc error: code = NotFound desc = could not find container \"1c03d0bd23716ac323f27de02a38f6945ce08cd004f1d6ecda9d79b4e62f025f\": container with ID starting with 1c03d0bd23716ac323f27de02a38f6945ce08cd004f1d6ecda9d79b4e62f025f not found: ID does not exist"
Aug 01 00:28:06.892095 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.892068    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"]
Aug 01 00:28:06.895865 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:06.895840    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheq26h7"]
Aug 01 00:28:07.946125 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:07.946093    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5e30159-1db1-4685-96b3-c363bf43e23a" path="/var/lib/kubelet/pods/f5e30159-1db1-4685-96b3-c363bf43e23a/volumes"
Aug 01 00:28:10.519939 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:10.519888    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:10.620212 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:10.620179    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:10.721792 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:10.721756    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:12.735906 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:12.735868    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:12.836453 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:12.836419    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:12.936740 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:12.936705    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:16.416850 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:16.416814    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:18.335663 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:18.335629    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:18.470524 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:18.470493    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:19.004225 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:19.004187    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:22.041596 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:22.041563    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:22.098551 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:22.098520    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:44.886448 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:44.886410    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:45.114104 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.114069    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"]
Aug 01 00:28:45.114568 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.114551    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e30159-1db1-4685-96b3-c363bf43e23a" containerName="main"
Aug 01 00:28:45.114615 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.114572    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e30159-1db1-4685-96b3-c363bf43e23a" containerName="main"
Aug 01 00:28:45.114664 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.114653    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5e30159-1db1-4685-96b3-c363bf43e23a" containerName="main"
Aug 01 00:28:45.117780 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.117757    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"
Aug 01 00:28:45.122324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.122126    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Aug 01 00:28:45.122324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.122158    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-7bstf\""
Aug 01 00:28:45.122324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.122167    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Aug 01 00:28:45.122324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.122179    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Aug 01 00:28:45.122324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.122126    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-kzfdd\""
Aug 01 00:28:45.124017 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.123987    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"]
Aug 01 00:28:45.138717 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.138643    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:45.183693 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.183654    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33ca6d1-6a4d-4fd1-a01d-f2f046d60638-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb\" (UID: \"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"
Aug 01 00:28:45.183877 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.183718    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lffh\" (UniqueName: \"kubernetes.io/projected/e33ca6d1-6a4d-4fd1-a01d-f2f046d60638-kube-api-access-9lffh\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb\" (UID: \"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"
Aug 01 00:28:45.285092 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.285052    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33ca6d1-6a4d-4fd1-a01d-f2f046d60638-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb\" (UID: \"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"
Aug 01 00:28:45.285297 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.285114    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lffh\" (UniqueName: \"kubernetes.io/projected/e33ca6d1-6a4d-4fd1-a01d-f2f046d60638-kube-api-access-9lffh\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb\" (UID: \"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"
Aug 01 00:28:45.287720 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.287694    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33ca6d1-6a4d-4fd1-a01d-f2f046d60638-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb\" (UID: \"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"
Aug 01 00:28:45.293667 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.293639    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lffh\" (UniqueName: \"kubernetes.io/projected/e33ca6d1-6a4d-4fd1-a01d-f2f046d60638-kube-api-access-9lffh\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb\" (UID: \"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"
Aug 01 00:28:45.430305 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.430212    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"
Aug 01 00:28:45.558975 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:45.558945    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"]
Aug 01 00:28:45.561750 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:28:45.561719    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33ca6d1_6a4d_4fd1_a01d_f2f046d60638.slice/crio-f122544323c9c5ecda73fa580cfbcf4752684445094de83fec0cadbc8164f79c WatchSource:0}: Error finding container f122544323c9c5ecda73fa580cfbcf4752684445094de83fec0cadbc8164f79c: Status 404 returned error can't find the container with id f122544323c9c5ecda73fa580cfbcf4752684445094de83fec0cadbc8164f79c
Aug 01 00:28:46.051410 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:46.051376    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb" event={"ID":"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638","Type":"ContainerStarted","Data":"e43ff9611760c21a4d4bcadf9f01c17b42f21135daa7be29c8003959d974f94b"}
Aug 01 00:28:46.051806 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:46.051424    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb" event={"ID":"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638","Type":"ContainerStarted","Data":"f122544323c9c5ecda73fa580cfbcf4752684445094de83fec0cadbc8164f79c"}
Aug 01 00:28:46.051806 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:46.051454    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"
Aug 01 00:28:46.067492 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:46.067428    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb" podStartSLOduration=1.067411911 podStartE2EDuration="1.067411911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:28:46.065654497 +0000 UTC m=+2568.759701911" watchObservedRunningTime="2026-08-01 00:28:46.067411911 +0000 UTC m=+2568.761459327"
Aug 01 00:28:48.259007 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:48.258969    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"]
Aug 01 00:28:48.267799 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:48.267765    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:48.348110 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:48.348072    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:49.385204 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:49.385159    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:49.486639 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:49.486578    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:49.599858 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:49.599812    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:53.713324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:53.713276    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:53.813324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:53.813283    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:53.913753 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:53.913721    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:54.466629 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:54.466566    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:58.069459 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:58.069428    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:58.236614 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:58.236579    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:58.338256 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:58.338180    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:59.489280 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:59.489239    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:59.588256 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:28:59.588217    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:00.992909 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:00.992867    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:01.228407 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.228375    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"]
Aug 01 00:29:01.232763 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.232741    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"
Aug 01 00:29:01.235675 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.235649    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Aug 01 00:29:01.235786 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.235683    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Aug 01 00:29:01.235786 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.235703    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-xjzmj\""
Aug 01 00:29:01.235786 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.235720    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Aug 01 00:29:01.235993 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.235819    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-qg6cx\""
Aug 01 00:29:01.238952 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.238933    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"]
Aug 01 00:29:01.275976 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.275858    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:01.328325 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.328286    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88714e6f-a845-4532-b7e9-1ff04eaec72a-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7\" (UID: \"88714e6f-a845-4532-b7e9-1ff04eaec72a\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"
Aug 01 00:29:01.328474 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.328362    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klpbc\" (UniqueName: \"kubernetes.io/projected/88714e6f-a845-4532-b7e9-1ff04eaec72a-kube-api-access-klpbc\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7\" (UID: \"88714e6f-a845-4532-b7e9-1ff04eaec72a\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"
Aug 01 00:29:01.429737 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.429696    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88714e6f-a845-4532-b7e9-1ff04eaec72a-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7\" (UID: \"88714e6f-a845-4532-b7e9-1ff04eaec72a\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"
Aug 01 00:29:01.429912 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.429760    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpbc\" (UniqueName: \"kubernetes.io/projected/88714e6f-a845-4532-b7e9-1ff04eaec72a-kube-api-access-klpbc\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7\" (UID: \"88714e6f-a845-4532-b7e9-1ff04eaec72a\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"
Aug 01 00:29:01.432251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.432228    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88714e6f-a845-4532-b7e9-1ff04eaec72a-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7\" (UID: \"88714e6f-a845-4532-b7e9-1ff04eaec72a\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"
Aug 01 00:29:01.439232 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.439208    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klpbc\" (UniqueName: \"kubernetes.io/projected/88714e6f-a845-4532-b7e9-1ff04eaec72a-kube-api-access-klpbc\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7\" (UID: \"88714e6f-a845-4532-b7e9-1ff04eaec72a\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"
Aug 01 00:29:01.544365 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.544280    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"
Aug 01 00:29:01.675929 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:01.675888    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"]
Aug 01 00:29:01.677851 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:29:01.677825    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88714e6f_a845_4532_b7e9_1ff04eaec72a.slice/crio-1e55634a655d4f0e21f5253be39719e09c174d6dfd99273ecf83ce705ba17902 WatchSource:0}: Error finding container 1e55634a655d4f0e21f5253be39719e09c174d6dfd99273ecf83ce705ba17902: Status 404 returned error can't find the container with id 1e55634a655d4f0e21f5253be39719e09c174d6dfd99273ecf83ce705ba17902
Aug 01 00:29:02.121550 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:02.121509    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7" event={"ID":"88714e6f-a845-4532-b7e9-1ff04eaec72a","Type":"ContainerStarted","Data":"d236c48fec0cf615752a0571c31898152d38ba9dd4372a2973a5fff4b4821a2b"}
Aug 01 00:29:02.122078 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:02.121563    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7" event={"ID":"88714e6f-a845-4532-b7e9-1ff04eaec72a","Type":"ContainerStarted","Data":"1e55634a655d4f0e21f5253be39719e09c174d6dfd99273ecf83ce705ba17902"}
Aug 01 00:29:02.122078 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:02.121640    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"
Aug 01 00:29:02.142973 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:02.142899    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7" podStartSLOduration=1.14288241 podStartE2EDuration="1.14288241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:02.140721768 +0000 UTC m=+2584.834769184" watchObservedRunningTime="2026-08-01 00:29:02.14288241 +0000 UTC m=+2584.836929826"
Aug 01 00:29:03.279384 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:03.279343    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb" podUID="e33ca6d1-6a4d-4fd1-a01d-f2f046d60638" containerName="main" containerID="cri-o://e43ff9611760c21a4d4bcadf9f01c17b42f21135daa7be29c8003959d974f94b" gracePeriod=45
Aug 01 00:29:03.528686 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:03.528661    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"
Aug 01 00:29:03.650069 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:03.649972    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33ca6d1-6a4d-4fd1-a01d-f2f046d60638-tls-certs\") pod \"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638\" (UID: \"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638\") "
Aug 01 00:29:03.650069 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:03.650037    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lffh\" (UniqueName: \"kubernetes.io/projected/e33ca6d1-6a4d-4fd1-a01d-f2f046d60638-kube-api-access-9lffh\") pod \"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638\" (UID: \"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638\") "
Aug 01 00:29:03.652253 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:03.652223    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33ca6d1-6a4d-4fd1-a01d-f2f046d60638-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e33ca6d1-6a4d-4fd1-a01d-f2f046d60638" (UID: "e33ca6d1-6a4d-4fd1-a01d-f2f046d60638"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:29:03.652382 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:03.652272    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33ca6d1-6a4d-4fd1-a01d-f2f046d60638-kube-api-access-9lffh" (OuterVolumeSpecName: "kube-api-access-9lffh") pod "e33ca6d1-6a4d-4fd1-a01d-f2f046d60638" (UID: "e33ca6d1-6a4d-4fd1-a01d-f2f046d60638"). InnerVolumeSpecName "kube-api-access-9lffh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:29:03.751606 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:03.751562    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33ca6d1-6a4d-4fd1-a01d-f2f046d60638-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:29:03.751606 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:03.751598    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lffh\" (UniqueName: \"kubernetes.io/projected/e33ca6d1-6a4d-4fd1-a01d-f2f046d60638-kube-api-access-9lffh\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:29:04.071481 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:04.071446    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"]
Aug 01 00:29:04.071664 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:04.071552    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:04.133189 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:04.133153    2551 generic.go:356] "Generic (PLEG): container finished" podID="e33ca6d1-6a4d-4fd1-a01d-f2f046d60638" containerID="e43ff9611760c21a4d4bcadf9f01c17b42f21135daa7be29c8003959d974f94b" exitCode=0
Aug 01 00:29:04.133189 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:04.133200    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"
Aug 01 00:29:04.133426 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:04.133242    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb" event={"ID":"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638","Type":"ContainerDied","Data":"e43ff9611760c21a4d4bcadf9f01c17b42f21135daa7be29c8003959d974f94b"}
Aug 01 00:29:04.133426 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:04.133304    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb" event={"ID":"e33ca6d1-6a4d-4fd1-a01d-f2f046d60638","Type":"ContainerDied","Data":"f122544323c9c5ecda73fa580cfbcf4752684445094de83fec0cadbc8164f79c"}
Aug 01 00:29:04.133426 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:04.133338    2551 scope.go:117] "RemoveContainer" containerID="e43ff9611760c21a4d4bcadf9f01c17b42f21135daa7be29c8003959d974f94b"
Aug 01 00:29:04.142738 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:04.142714    2551 scope.go:117] "RemoveContainer" containerID="e43ff9611760c21a4d4bcadf9f01c17b42f21135daa7be29c8003959d974f94b"
Aug 01 00:29:04.143007 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:29:04.142987    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e43ff9611760c21a4d4bcadf9f01c17b42f21135daa7be29c8003959d974f94b\": container with ID starting with e43ff9611760c21a4d4bcadf9f01c17b42f21135daa7be29c8003959d974f94b not found: ID does not exist" containerID="e43ff9611760c21a4d4bcadf9f01c17b42f21135daa7be29c8003959d974f94b"
Aug 01 00:29:04.143074 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:04.143015    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e43ff9611760c21a4d4bcadf9f01c17b42f21135daa7be29c8003959d974f94b"} err="failed to get container status \"e43ff9611760c21a4d4bcadf9f01c17b42f21135daa7be29c8003959d974f94b\": rpc error: code = NotFound desc = could not find container \"e43ff9611760c21a4d4bcadf9f01c17b42f21135daa7be29c8003959d974f94b\": container with ID starting with e43ff9611760c21a4d4bcadf9f01c17b42f21135daa7be29c8003959d974f94b not found: ID does not exist"
Aug 01 00:29:04.157107 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:04.157080    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:04.166440 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:04.166415    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"]
Aug 01 00:29:04.170311 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:04.170288    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-69thb"]
Aug 01 00:29:05.946657 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:05.946621    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e33ca6d1-6a4d-4fd1-a01d-f2f046d60638" path="/var/lib/kubelet/pods/e33ca6d1-6a4d-4fd1-a01d-f2f046d60638/volumes"
Aug 01 00:29:06.593345 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:06.593313    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:06.776632 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:06.776593    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:09.670409 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:09.670367    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:11.666610 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:11.666574    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:19.153117 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:19.153071    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7" podUID="88714e6f-a845-4532-b7e9-1ff04eaec72a" containerName="main" containerID="cri-o://d236c48fec0cf615752a0571c31898152d38ba9dd4372a2973a5fff4b4821a2b" gracePeriod=45
Aug 01 00:29:19.411202 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:19.411139    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"
Aug 01 00:29:19.489469 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:19.489429    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klpbc\" (UniqueName: \"kubernetes.io/projected/88714e6f-a845-4532-b7e9-1ff04eaec72a-kube-api-access-klpbc\") pod \"88714e6f-a845-4532-b7e9-1ff04eaec72a\" (UID: \"88714e6f-a845-4532-b7e9-1ff04eaec72a\") "
Aug 01 00:29:19.489652 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:19.489512    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88714e6f-a845-4532-b7e9-1ff04eaec72a-tls-certs\") pod \"88714e6f-a845-4532-b7e9-1ff04eaec72a\" (UID: \"88714e6f-a845-4532-b7e9-1ff04eaec72a\") "
Aug 01 00:29:19.491755 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:19.491715    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88714e6f-a845-4532-b7e9-1ff04eaec72a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88714e6f-a845-4532-b7e9-1ff04eaec72a" (UID: "88714e6f-a845-4532-b7e9-1ff04eaec72a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:29:19.491755 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:19.491741    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88714e6f-a845-4532-b7e9-1ff04eaec72a-kube-api-access-klpbc" (OuterVolumeSpecName: "kube-api-access-klpbc") pod "88714e6f-a845-4532-b7e9-1ff04eaec72a" (UID: "88714e6f-a845-4532-b7e9-1ff04eaec72a"). InnerVolumeSpecName "kube-api-access-klpbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:29:19.591082 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:19.591044    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klpbc\" (UniqueName: \"kubernetes.io/projected/88714e6f-a845-4532-b7e9-1ff04eaec72a-kube-api-access-klpbc\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:29:19.591082 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:19.591076    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88714e6f-a845-4532-b7e9-1ff04eaec72a-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:29:20.214379 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.214261    2551 generic.go:356] "Generic (PLEG): container finished" podID="88714e6f-a845-4532-b7e9-1ff04eaec72a" containerID="d236c48fec0cf615752a0571c31898152d38ba9dd4372a2973a5fff4b4821a2b" exitCode=0
Aug 01 00:29:20.214379 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.214316    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"
Aug 01 00:29:20.214379 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.214346    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7" event={"ID":"88714e6f-a845-4532-b7e9-1ff04eaec72a","Type":"ContainerDied","Data":"d236c48fec0cf615752a0571c31898152d38ba9dd4372a2973a5fff4b4821a2b"}
Aug 01 00:29:20.214905 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.214397    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7" event={"ID":"88714e6f-a845-4532-b7e9-1ff04eaec72a","Type":"ContainerDied","Data":"1e55634a655d4f0e21f5253be39719e09c174d6dfd99273ecf83ce705ba17902"}
Aug 01 00:29:20.214905 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.214423    2551 scope.go:117] "RemoveContainer" containerID="d236c48fec0cf615752a0571c31898152d38ba9dd4372a2973a5fff4b4821a2b"
Aug 01 00:29:20.222580 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.222560    2551 scope.go:117] "RemoveContainer" containerID="d236c48fec0cf615752a0571c31898152d38ba9dd4372a2973a5fff4b4821a2b"
Aug 01 00:29:20.222814 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:29:20.222794    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d236c48fec0cf615752a0571c31898152d38ba9dd4372a2973a5fff4b4821a2b\": container with ID starting with d236c48fec0cf615752a0571c31898152d38ba9dd4372a2973a5fff4b4821a2b not found: ID does not exist" containerID="d236c48fec0cf615752a0571c31898152d38ba9dd4372a2973a5fff4b4821a2b"
Aug 01 00:29:20.222872 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.222821    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d236c48fec0cf615752a0571c31898152d38ba9dd4372a2973a5fff4b4821a2b"} err="failed to get container status \"d236c48fec0cf615752a0571c31898152d38ba9dd4372a2973a5fff4b4821a2b\": rpc error: code = NotFound desc = could not find container \"d236c48fec0cf615752a0571c31898152d38ba9dd4372a2973a5fff4b4821a2b\": container with ID starting with d236c48fec0cf615752a0571c31898152d38ba9dd4372a2973a5fff4b4821a2b not found: ID does not exist"
Aug 01 00:29:20.228893 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.228867    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"]
Aug 01 00:29:20.232087 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.232066    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-6d99c88894w69w7"]
Aug 01 00:29:20.401265 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.401230    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:20.624227 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.624192    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"]
Aug 01 00:29:20.625130 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.625105    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33ca6d1-6a4d-4fd1-a01d-f2f046d60638" containerName="main"
Aug 01 00:29:20.625276 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.625265    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33ca6d1-6a4d-4fd1-a01d-f2f046d60638" containerName="main"
Aug 01 00:29:20.625373 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.625363    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88714e6f-a845-4532-b7e9-1ff04eaec72a" containerName="main"
Aug 01 00:29:20.625453 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.625444    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="88714e6f-a845-4532-b7e9-1ff04eaec72a" containerName="main"
Aug 01 00:29:20.625722 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.625710    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33ca6d1-6a4d-4fd1-a01d-f2f046d60638" containerName="main"
Aug 01 00:29:20.625814 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.625803    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="88714e6f-a845-4532-b7e9-1ff04eaec72a" containerName="main"
Aug 01 00:29:20.631285 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.631262    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"
Aug 01 00:29:20.633178 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.633147    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"]
Aug 01 00:29:20.634271 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.634169    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Aug 01 00:29:20.634366 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.634315    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Aug 01 00:29:20.635154 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.635128    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Aug 01 00:29:20.635256 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.634889    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-szl5p\""
Aug 01 00:29:20.635256 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.635197    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-tkj85\""
Aug 01 00:29:20.653442 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.653406    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:20.700103 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.700066    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedaa21f-6a2e-4020-aa0f-a5d7036e3eae-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls\" (UID: \"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"
Aug 01 00:29:20.700267 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.700205    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f766x\" (UniqueName: \"kubernetes.io/projected/bedaa21f-6a2e-4020-aa0f-a5d7036e3eae-kube-api-access-f766x\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls\" (UID: \"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"
Aug 01 00:29:20.801121 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.801082    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f766x\" (UniqueName: \"kubernetes.io/projected/bedaa21f-6a2e-4020-aa0f-a5d7036e3eae-kube-api-access-f766x\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls\" (UID: \"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"
Aug 01 00:29:20.801305 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.801171    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedaa21f-6a2e-4020-aa0f-a5d7036e3eae-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls\" (UID: \"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"
Aug 01 00:29:20.803742 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.803709    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedaa21f-6a2e-4020-aa0f-a5d7036e3eae-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls\" (UID: \"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"
Aug 01 00:29:20.808987 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.808958    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f766x\" (UniqueName: \"kubernetes.io/projected/bedaa21f-6a2e-4020-aa0f-a5d7036e3eae-kube-api-access-f766x\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls\" (UID: \"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"
Aug 01 00:29:20.943841 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:20.943756    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"
Aug 01 00:29:21.073287 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:21.073252    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"]
Aug 01 00:29:21.076270 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:29:21.076244    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedaa21f_6a2e_4020_aa0f_a5d7036e3eae.slice/crio-a236315c95f5426d6e0dd568a29100acc3260cdf3dae165799cb743bb2961263 WatchSource:0}: Error finding container a236315c95f5426d6e0dd568a29100acc3260cdf3dae165799cb743bb2961263: Status 404 returned error can't find the container with id a236315c95f5426d6e0dd568a29100acc3260cdf3dae165799cb743bb2961263
Aug 01 00:29:21.221802 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:21.221771    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls" event={"ID":"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae","Type":"ContainerStarted","Data":"2506902cac55bbc729ac0021fa1f011679415a322954c43952f340246c6b95d1"}
Aug 01 00:29:21.222238 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:21.221825    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls" event={"ID":"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae","Type":"ContainerStarted","Data":"a236315c95f5426d6e0dd568a29100acc3260cdf3dae165799cb743bb2961263"}
Aug 01 00:29:21.222238 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:21.221873    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"
Aug 01 00:29:21.237587 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:21.237543    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls" podStartSLOduration=1.237529972 podStartE2EDuration="1.237529972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:21.236453433 +0000 UTC m=+2603.930500889" watchObservedRunningTime="2026-08-01 00:29:21.237529972 +0000 UTC m=+2603.931577387"
Aug 01 00:29:21.947894 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:21.947862    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88714e6f-a845-4532-b7e9-1ff04eaec72a" path="/var/lib/kubelet/pods/88714e6f-a845-4532-b7e9-1ff04eaec72a/volumes"
Aug 01 00:29:29.724483 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:29.724434    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:31.776207 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:31.776171    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:48.406993 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:48.406910    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:52.231403 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:52.231371    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"
Aug 01 00:29:55.088951 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:29:55.088905    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:00.083401 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:00.083328    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:00.182300 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:00.182267    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:05.026135 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:05.026101    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"]
Aug 01 00:30:05.026581 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:05.026296    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t" podUID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerName="istio-proxy" containerID="cri-o://33e4190d16ff75b5f59f59e38cc7514431cd21c4aa64fed9c23409d816a4da10" gracePeriod=30
Aug 01 00:30:05.065844 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:05.065811    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:10.271654 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.271627    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:30:10.349715 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.349688    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2c0880-fd5c-4872-bacc-652bfdd61b39-istiod-ca-cert\") pod \"be2c0880-fd5c-4872-bacc-652bfdd61b39\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") "
Aug 01 00:30:10.349826 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.349725    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-workload-socket\") pod \"be2c0880-fd5c-4872-bacc-652bfdd61b39\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") "
Aug 01 00:30:10.349826 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.349764    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-workload-certs\") pod \"be2c0880-fd5c-4872-bacc-652bfdd61b39\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") "
Aug 01 00:30:10.349826 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.349779    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-envoy\") pod \"be2c0880-fd5c-4872-bacc-652bfdd61b39\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") "
Aug 01 00:30:10.350014 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.349832    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dn4j\" (UniqueName: \"kubernetes.io/projected/be2c0880-fd5c-4872-bacc-652bfdd61b39-kube-api-access-6dn4j\") pod \"be2c0880-fd5c-4872-bacc-652bfdd61b39\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") "
Aug 01 00:30:10.350014 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.349858    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-data\") pod \"be2c0880-fd5c-4872-bacc-652bfdd61b39\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") "
Aug 01 00:30:10.350014 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.349905    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-podinfo\") pod \"be2c0880-fd5c-4872-bacc-652bfdd61b39\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") "
Aug 01 00:30:10.350014 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.349962    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-token\") pod \"be2c0880-fd5c-4872-bacc-652bfdd61b39\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") "
Aug 01 00:30:10.350014 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.350000    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-credential-socket\") pod \"be2c0880-fd5c-4872-bacc-652bfdd61b39\" (UID: \"be2c0880-fd5c-4872-bacc-652bfdd61b39\") "
Aug 01 00:30:10.350263 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.350120    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2c0880-fd5c-4872-bacc-652bfdd61b39-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "be2c0880-fd5c-4872-bacc-652bfdd61b39" (UID: "be2c0880-fd5c-4872-bacc-652bfdd61b39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:30:10.350263 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.350236    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "be2c0880-fd5c-4872-bacc-652bfdd61b39" (UID: "be2c0880-fd5c-4872-bacc-652bfdd61b39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:30:10.350374 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.350255    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "be2c0880-fd5c-4872-bacc-652bfdd61b39" (UID: "be2c0880-fd5c-4872-bacc-652bfdd61b39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:30:10.350434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.350378    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2c0880-fd5c-4872-bacc-652bfdd61b39-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:30:10.350434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.350381    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-data" (OuterVolumeSpecName: "istio-data") pod "be2c0880-fd5c-4872-bacc-652bfdd61b39" (UID: "be2c0880-fd5c-4872-bacc-652bfdd61b39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:30:10.350434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.350399    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:30:10.350434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.350413    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:30:10.350639 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.350613    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "be2c0880-fd5c-4872-bacc-652bfdd61b39" (UID: "be2c0880-fd5c-4872-bacc-652bfdd61b39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:30:10.352335 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.352302    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "be2c0880-fd5c-4872-bacc-652bfdd61b39" (UID: "be2c0880-fd5c-4872-bacc-652bfdd61b39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:30:10.352459 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.352373    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2c0880-fd5c-4872-bacc-652bfdd61b39-kube-api-access-6dn4j" (OuterVolumeSpecName: "kube-api-access-6dn4j") pod "be2c0880-fd5c-4872-bacc-652bfdd61b39" (UID: "be2c0880-fd5c-4872-bacc-652bfdd61b39"). InnerVolumeSpecName "kube-api-access-6dn4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:30:10.352459 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.352407    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "be2c0880-fd5c-4872-bacc-652bfdd61b39" (UID: "be2c0880-fd5c-4872-bacc-652bfdd61b39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 00:30:10.352632 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.352610    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-token" (OuterVolumeSpecName: "istio-token") pod "be2c0880-fd5c-4872-bacc-652bfdd61b39" (UID: "be2c0880-fd5c-4872-bacc-652bfdd61b39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:30:10.442159 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.442125    2551 generic.go:356] "Generic (PLEG): container finished" podID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerID="33e4190d16ff75b5f59f59e38cc7514431cd21c4aa64fed9c23409d816a4da10" exitCode=0
Aug 01 00:30:10.442330 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.442183    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"
Aug 01 00:30:10.442330 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.442213    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t" event={"ID":"be2c0880-fd5c-4872-bacc-652bfdd61b39","Type":"ContainerDied","Data":"33e4190d16ff75b5f59f59e38cc7514431cd21c4aa64fed9c23409d816a4da10"}
Aug 01 00:30:10.442330 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.442298    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t" event={"ID":"be2c0880-fd5c-4872-bacc-652bfdd61b39","Type":"ContainerDied","Data":"bb5af7f5226b672466ef64795009abeb3a01688ae69f98e219d9a122b3675247"}
Aug 01 00:30:10.442330 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.442321    2551 scope.go:117] "RemoveContainer" containerID="33e4190d16ff75b5f59f59e38cc7514431cd21c4aa64fed9c23409d816a4da10"
Aug 01 00:30:10.450824 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.450802    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:30:10.450824 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.450824    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:30:10.450997 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.450836    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dn4j\" (UniqueName: \"kubernetes.io/projected/be2c0880-fd5c-4872-bacc-652bfdd61b39-kube-api-access-6dn4j\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:30:10.450997 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.450845    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:30:10.450997 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.450853    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:30:10.450997 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.450861    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2c0880-fd5c-4872-bacc-652bfdd61b39-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:30:10.451183 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.451002    2551 scope.go:117] "RemoveContainer" containerID="0fa8a9ecbe6e0b0963979db201bf0e36c5ed764865fe99223fb45a89a01a1a54"
Aug 01 00:30:10.458335 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.458319    2551 scope.go:117] "RemoveContainer" containerID="33e4190d16ff75b5f59f59e38cc7514431cd21c4aa64fed9c23409d816a4da10"
Aug 01 00:30:10.458561 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:30:10.458540    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e4190d16ff75b5f59f59e38cc7514431cd21c4aa64fed9c23409d816a4da10\": container with ID starting with 33e4190d16ff75b5f59f59e38cc7514431cd21c4aa64fed9c23409d816a4da10 not found: ID does not exist" containerID="33e4190d16ff75b5f59f59e38cc7514431cd21c4aa64fed9c23409d816a4da10"
Aug 01 00:30:10.458604 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.458570    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e4190d16ff75b5f59f59e38cc7514431cd21c4aa64fed9c23409d816a4da10"} err="failed to get container status \"33e4190d16ff75b5f59f59e38cc7514431cd21c4aa64fed9c23409d816a4da10\": rpc error: code = NotFound desc = could not find container \"33e4190d16ff75b5f59f59e38cc7514431cd21c4aa64fed9c23409d816a4da10\": container with ID starting with 33e4190d16ff75b5f59f59e38cc7514431cd21c4aa64fed9c23409d816a4da10 not found: ID does not exist"
Aug 01 00:30:10.458604 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.458588    2551 scope.go:117] "RemoveContainer" containerID="0fa8a9ecbe6e0b0963979db201bf0e36c5ed764865fe99223fb45a89a01a1a54"
Aug 01 00:30:10.458797 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:30:10.458781    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa8a9ecbe6e0b0963979db201bf0e36c5ed764865fe99223fb45a89a01a1a54\": container with ID starting with 0fa8a9ecbe6e0b0963979db201bf0e36c5ed764865fe99223fb45a89a01a1a54 not found: ID does not exist" containerID="0fa8a9ecbe6e0b0963979db201bf0e36c5ed764865fe99223fb45a89a01a1a54"
Aug 01 00:30:10.458834 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.458805    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa8a9ecbe6e0b0963979db201bf0e36c5ed764865fe99223fb45a89a01a1a54"} err="failed to get container status \"0fa8a9ecbe6e0b0963979db201bf0e36c5ed764865fe99223fb45a89a01a1a54\": rpc error: code = NotFound desc = could not find container \"0fa8a9ecbe6e0b0963979db201bf0e36c5ed764865fe99223fb45a89a01a1a54\": container with ID starting with 0fa8a9ecbe6e0b0963979db201bf0e36c5ed764865fe99223fb45a89a01a1a54 not found: ID does not exist"
Aug 01 00:30:10.462925 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.462894    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"]
Aug 01 00:30:10.466562 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.466540    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-jhv6t"]
Aug 01 00:30:10.661243 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.661159    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:10.837453 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.837422    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"]
Aug 01 00:30:10.837747 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.837734    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerName="istio-proxy"
Aug 01 00:30:10.837805 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.837749    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerName="istio-proxy"
Aug 01 00:30:10.837805 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.837767    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerName="istio-proxy"
Aug 01 00:30:10.837805 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.837772    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerName="istio-proxy"
Aug 01 00:30:10.837903 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.837825    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerName="istio-proxy"
Aug 01 00:30:10.837903 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.837834    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="be2c0880-fd5c-4872-bacc-652bfdd61b39" containerName="istio-proxy"
Aug 01 00:30:10.842217 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.842197    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"
Aug 01 00:30:10.844485 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.844455    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Aug 01 00:30:10.844605 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.844504    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-fgvz5\""
Aug 01 00:30:10.844679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.844630    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-vssjt\""
Aug 01 00:30:10.844814 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.844780    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Aug 01 00:30:10.844951 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.844932    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Aug 01 00:30:10.849814 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.849792    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"]
Aug 01 00:30:10.872466 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.872433    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:10.955440 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.955356    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b8e62d3-1b92-497f-97d9-c5efa62536f6-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78\" (UID: \"7b8e62d3-1b92-497f-97d9-c5efa62536f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"
Aug 01 00:30:10.955598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:10.955487    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966zk\" (UniqueName: \"kubernetes.io/projected/7b8e62d3-1b92-497f-97d9-c5efa62536f6-kube-api-access-966zk\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78\" (UID: \"7b8e62d3-1b92-497f-97d9-c5efa62536f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"
Aug 01 00:30:11.056782 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:11.056742    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966zk\" (UniqueName: \"kubernetes.io/projected/7b8e62d3-1b92-497f-97d9-c5efa62536f6-kube-api-access-966zk\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78\" (UID: \"7b8e62d3-1b92-497f-97d9-c5efa62536f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"
Aug 01 00:30:11.056976 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:11.056792    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b8e62d3-1b92-497f-97d9-c5efa62536f6-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78\" (UID: \"7b8e62d3-1b92-497f-97d9-c5efa62536f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"
Aug 01 00:30:11.059302 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:11.059281    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b8e62d3-1b92-497f-97d9-c5efa62536f6-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78\" (UID: \"7b8e62d3-1b92-497f-97d9-c5efa62536f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"
Aug 01 00:30:11.064098 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:11.064071    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-966zk\" (UniqueName: \"kubernetes.io/projected/7b8e62d3-1b92-497f-97d9-c5efa62536f6-kube-api-access-966zk\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78\" (UID: \"7b8e62d3-1b92-497f-97d9-c5efa62536f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"
Aug 01 00:30:11.154655 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:11.154595    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"
Aug 01 00:30:11.283166 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:11.283137    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"]
Aug 01 00:30:11.285544 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:30:11.285518    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8e62d3_1b92_497f_97d9_c5efa62536f6.slice/crio-17c5759ae8b7cdabce71b0249e233bc17e1ab922884e4e765cb725dff26a06e0 WatchSource:0}: Error finding container 17c5759ae8b7cdabce71b0249e233bc17e1ab922884e4e765cb725dff26a06e0: Status 404 returned error can't find the container with id 17c5759ae8b7cdabce71b0249e233bc17e1ab922884e4e765cb725dff26a06e0
Aug 01 00:30:11.448399 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:11.448357    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78" event={"ID":"7b8e62d3-1b92-497f-97d9-c5efa62536f6","Type":"ContainerStarted","Data":"7902428de96a11c48a3cd2c538aca67dff9c949a109ec1e1865c69c729cff908"}
Aug 01 00:30:11.448604 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:11.448425    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78" event={"ID":"7b8e62d3-1b92-497f-97d9-c5efa62536f6","Type":"ContainerStarted","Data":"17c5759ae8b7cdabce71b0249e233bc17e1ab922884e4e765cb725dff26a06e0"}
Aug 01 00:30:11.448604 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:11.448464    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"
Aug 01 00:30:11.464460 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:11.464415    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78" podStartSLOduration=1.4644001 podStartE2EDuration="1.4644001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:11.462630377 +0000 UTC m=+2654.156677794" watchObservedRunningTime="2026-08-01 00:30:11.4644001 +0000 UTC m=+2654.158447516"
Aug 01 00:30:11.947349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:11.947319    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be2c0880-fd5c-4872-bacc-652bfdd61b39" path="/var/lib/kubelet/pods/be2c0880-fd5c-4872-bacc-652bfdd61b39/volumes"
Aug 01 00:30:22.852786 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:22.852749    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:26.364101 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:26.364047    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:27.143730 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:27.143688    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:32.695718 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:32.695683    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:42.456214 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:42.456182    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"
Aug 01 00:30:47.895077 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:47.895037    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:55.844346 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:55.844256    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:55.946001 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:30:55.945966    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:32.266478 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:32.266420    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:32.521477 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:32.521379    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"]
Aug 01 00:31:32.637215 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:32.637179    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:32.750477 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:32.750442    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:42.230581 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:42.230541    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls" podUID="bedaa21f-6a2e-4020-aa0f-a5d7036e3eae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:31:47.540878 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.540811    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls" podUID="bedaa21f-6a2e-4020-aa0f-a5d7036e3eae" containerName="main" containerID="cri-o://2506902cac55bbc729ac0021fa1f011679415a322954c43952f340246c6b95d1" gracePeriod=45
Aug 01 00:31:47.794634 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.794569    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"
Aug 01 00:31:47.881626 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.881594    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f766x\" (UniqueName: \"kubernetes.io/projected/bedaa21f-6a2e-4020-aa0f-a5d7036e3eae-kube-api-access-f766x\") pod \"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae\" (UID: \"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae\") "
Aug 01 00:31:47.881837 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.881648    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedaa21f-6a2e-4020-aa0f-a5d7036e3eae-tls-certs\") pod \"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae\" (UID: \"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae\") "
Aug 01 00:31:47.881837 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.881716    2551 generic.go:356] "Generic (PLEG): container finished" podID="bedaa21f-6a2e-4020-aa0f-a5d7036e3eae" containerID="2506902cac55bbc729ac0021fa1f011679415a322954c43952f340246c6b95d1" exitCode=0
Aug 01 00:31:47.881837 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.881782    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls" event={"ID":"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae","Type":"ContainerDied","Data":"2506902cac55bbc729ac0021fa1f011679415a322954c43952f340246c6b95d1"}
Aug 01 00:31:47.881837 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.881818    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"
Aug 01 00:31:47.882081 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.881856    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls" event={"ID":"bedaa21f-6a2e-4020-aa0f-a5d7036e3eae","Type":"ContainerDied","Data":"a236315c95f5426d6e0dd568a29100acc3260cdf3dae165799cb743bb2961263"}
Aug 01 00:31:47.882081 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.881889    2551 scope.go:117] "RemoveContainer" containerID="2506902cac55bbc729ac0021fa1f011679415a322954c43952f340246c6b95d1"
Aug 01 00:31:47.884098 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.884067    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bedaa21f-6a2e-4020-aa0f-a5d7036e3eae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bedaa21f-6a2e-4020-aa0f-a5d7036e3eae" (UID: "bedaa21f-6a2e-4020-aa0f-a5d7036e3eae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:31:47.884098 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.884088    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedaa21f-6a2e-4020-aa0f-a5d7036e3eae-kube-api-access-f766x" (OuterVolumeSpecName: "kube-api-access-f766x") pod "bedaa21f-6a2e-4020-aa0f-a5d7036e3eae" (UID: "bedaa21f-6a2e-4020-aa0f-a5d7036e3eae"). InnerVolumeSpecName "kube-api-access-f766x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:31:47.894489 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.894472    2551 scope.go:117] "RemoveContainer" containerID="2506902cac55bbc729ac0021fa1f011679415a322954c43952f340246c6b95d1"
Aug 01 00:31:47.894750 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:31:47.894731    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2506902cac55bbc729ac0021fa1f011679415a322954c43952f340246c6b95d1\": container with ID starting with 2506902cac55bbc729ac0021fa1f011679415a322954c43952f340246c6b95d1 not found: ID does not exist" containerID="2506902cac55bbc729ac0021fa1f011679415a322954c43952f340246c6b95d1"
Aug 01 00:31:47.894801 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.894758    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2506902cac55bbc729ac0021fa1f011679415a322954c43952f340246c6b95d1"} err="failed to get container status \"2506902cac55bbc729ac0021fa1f011679415a322954c43952f340246c6b95d1\": rpc error: code = NotFound desc = could not find container \"2506902cac55bbc729ac0021fa1f011679415a322954c43952f340246c6b95d1\": container with ID starting with 2506902cac55bbc729ac0021fa1f011679415a322954c43952f340246c6b95d1 not found: ID does not exist"
Aug 01 00:31:47.982592 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.982555    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f766x\" (UniqueName: \"kubernetes.io/projected/bedaa21f-6a2e-4020-aa0f-a5d7036e3eae-kube-api-access-f766x\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:31:47.982592 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:47.982586    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedaa21f-6a2e-4020-aa0f-a5d7036e3eae-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:31:48.196600 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.196477    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"]
Aug 01 00:31:48.198101 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.198078    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vdbls"]
Aug 01 00:31:48.704013 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.703977    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:48.847698 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.847661    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"]
Aug 01 00:31:48.848230 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.848208    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bedaa21f-6a2e-4020-aa0f-a5d7036e3eae" containerName="main"
Aug 01 00:31:48.848345 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.848234    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="bedaa21f-6a2e-4020-aa0f-a5d7036e3eae" containerName="main"
Aug 01 00:31:48.848345 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.848335    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="bedaa21f-6a2e-4020-aa0f-a5d7036e3eae" containerName="main"
Aug 01 00:31:48.853483 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.853460    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"
Aug 01 00:31:48.856328 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.856305    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-qddjs\""
Aug 01 00:31:48.856446 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.856334    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Aug 01 00:31:48.856446 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.856347    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Aug 01 00:31:48.856446 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.856305    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Aug 01 00:31:48.856620 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.856305    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-tkj85\""
Aug 01 00:31:48.860828 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.860758    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"]
Aug 01 00:31:48.889813 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.889780    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c022e2-6746-4845-9149-c87b72ea42bc-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2\" (UID: \"d5c022e2-6746-4845-9149-c87b72ea42bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"
Aug 01 00:31:48.889965 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.889842    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlq2c\" (UniqueName: \"kubernetes.io/projected/d5c022e2-6746-4845-9149-c87b72ea42bc-kube-api-access-xlq2c\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2\" (UID: \"d5c022e2-6746-4845-9149-c87b72ea42bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"
Aug 01 00:31:48.950104 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.950045    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:48.991262 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.991218    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c022e2-6746-4845-9149-c87b72ea42bc-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2\" (UID: \"d5c022e2-6746-4845-9149-c87b72ea42bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"
Aug 01 00:31:48.991441 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.991383    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlq2c\" (UniqueName: \"kubernetes.io/projected/d5c022e2-6746-4845-9149-c87b72ea42bc-kube-api-access-xlq2c\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2\" (UID: \"d5c022e2-6746-4845-9149-c87b72ea42bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"
Aug 01 00:31:48.993988 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.993965    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c022e2-6746-4845-9149-c87b72ea42bc-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2\" (UID: \"d5c022e2-6746-4845-9149-c87b72ea42bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"
Aug 01 00:31:48.998752 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:48.998726    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlq2c\" (UniqueName: \"kubernetes.io/projected/d5c022e2-6746-4845-9149-c87b72ea42bc-kube-api-access-xlq2c\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2\" (UID: \"d5c022e2-6746-4845-9149-c87b72ea42bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"
Aug 01 00:31:49.165845 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:49.165805    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"
Aug 01 00:31:49.298826 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:49.298789    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"]
Aug 01 00:31:49.302135 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:31:49.302094    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c022e2_6746_4845_9149_c87b72ea42bc.slice/crio-395bdde30ee958fd4f320b5d51d212f321e95cec1de1820094b845b1564e5bba WatchSource:0}: Error finding container 395bdde30ee958fd4f320b5d51d212f321e95cec1de1820094b845b1564e5bba: Status 404 returned error can't find the container with id 395bdde30ee958fd4f320b5d51d212f321e95cec1de1820094b845b1564e5bba
Aug 01 00:31:49.893620 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:49.893589    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2" event={"ID":"d5c022e2-6746-4845-9149-c87b72ea42bc","Type":"ContainerStarted","Data":"bb195adf75c0a96f3affba9a7d0c06d26b295f11814ab1ef55614d35afed1538"}
Aug 01 00:31:49.893999 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:49.893639    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2" event={"ID":"d5c022e2-6746-4845-9149-c87b72ea42bc","Type":"ContainerStarted","Data":"395bdde30ee958fd4f320b5d51d212f321e95cec1de1820094b845b1564e5bba"}
Aug 01 00:31:49.893999 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:49.893682    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"
Aug 01 00:31:49.909777 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:49.909726    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2" podStartSLOduration=1.9097124779999999 podStartE2EDuration="1.909712478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:31:49.908604493 +0000 UTC m=+2752.602651909" watchObservedRunningTime="2026-08-01 00:31:49.909712478 +0000 UTC m=+2752.603759893"
Aug 01 00:31:49.946580 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:49.946549    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bedaa21f-6a2e-4020-aa0f-a5d7036e3eae" path="/var/lib/kubelet/pods/bedaa21f-6a2e-4020-aa0f-a5d7036e3eae/volumes"
Aug 01 00:31:57.898217 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:31:57.898182    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:15.285476 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:15.285436    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"]
Aug 01 00:32:15.293492 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:15.293457    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:15.352117 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:15.352066    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:20.900628 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:20.900547    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"
Aug 01 00:32:22.456898 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:22.456857    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78" podUID="7b8e62d3-1b92-497f-97d9-c5efa62536f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:32:30.307715 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:30.307672    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78" podUID="7b8e62d3-1b92-497f-97d9-c5efa62536f6" containerName="main" containerID="cri-o://7902428de96a11c48a3cd2c538aca67dff9c949a109ec1e1865c69c729cff908" gracePeriod=45
Aug 01 00:32:30.568304 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:30.568240    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"
Aug 01 00:32:30.661207 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:30.661169    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-966zk\" (UniqueName: \"kubernetes.io/projected/7b8e62d3-1b92-497f-97d9-c5efa62536f6-kube-api-access-966zk\") pod \"7b8e62d3-1b92-497f-97d9-c5efa62536f6\" (UID: \"7b8e62d3-1b92-497f-97d9-c5efa62536f6\") "
Aug 01 00:32:30.661394 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:30.661220    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b8e62d3-1b92-497f-97d9-c5efa62536f6-tls-certs\") pod \"7b8e62d3-1b92-497f-97d9-c5efa62536f6\" (UID: \"7b8e62d3-1b92-497f-97d9-c5efa62536f6\") "
Aug 01 00:32:30.663580 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:30.663545    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b8e62d3-1b92-497f-97d9-c5efa62536f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b8e62d3-1b92-497f-97d9-c5efa62536f6" (UID: "7b8e62d3-1b92-497f-97d9-c5efa62536f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:32:30.663717 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:30.663628    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8e62d3-1b92-497f-97d9-c5efa62536f6-kube-api-access-966zk" (OuterVolumeSpecName: "kube-api-access-966zk") pod "7b8e62d3-1b92-497f-97d9-c5efa62536f6" (UID: "7b8e62d3-1b92-497f-97d9-c5efa62536f6"). InnerVolumeSpecName "kube-api-access-966zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:32:30.762206 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:30.762167    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-966zk\" (UniqueName: \"kubernetes.io/projected/7b8e62d3-1b92-497f-97d9-c5efa62536f6-kube-api-access-966zk\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:32:30.762206 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:30.762201    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b8e62d3-1b92-497f-97d9-c5efa62536f6-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:32:31.079075 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:31.079041    2551 generic.go:356] "Generic (PLEG): container finished" podID="7b8e62d3-1b92-497f-97d9-c5efa62536f6" containerID="7902428de96a11c48a3cd2c538aca67dff9c949a109ec1e1865c69c729cff908" exitCode=0
Aug 01 00:32:31.079251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:31.079090    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"
Aug 01 00:32:31.079251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:31.079114    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78" event={"ID":"7b8e62d3-1b92-497f-97d9-c5efa62536f6","Type":"ContainerDied","Data":"7902428de96a11c48a3cd2c538aca67dff9c949a109ec1e1865c69c729cff908"}
Aug 01 00:32:31.079251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:31.079158    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78" event={"ID":"7b8e62d3-1b92-497f-97d9-c5efa62536f6","Type":"ContainerDied","Data":"17c5759ae8b7cdabce71b0249e233bc17e1ab922884e4e765cb725dff26a06e0"}
Aug 01 00:32:31.079251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:31.079180    2551 scope.go:117] "RemoveContainer" containerID="7902428de96a11c48a3cd2c538aca67dff9c949a109ec1e1865c69c729cff908"
Aug 01 00:32:31.087571 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:31.087551    2551 scope.go:117] "RemoveContainer" containerID="7902428de96a11c48a3cd2c538aca67dff9c949a109ec1e1865c69c729cff908"
Aug 01 00:32:31.087934 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:32:31.087886    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7902428de96a11c48a3cd2c538aca67dff9c949a109ec1e1865c69c729cff908\": container with ID starting with 7902428de96a11c48a3cd2c538aca67dff9c949a109ec1e1865c69c729cff908 not found: ID does not exist" containerID="7902428de96a11c48a3cd2c538aca67dff9c949a109ec1e1865c69c729cff908"
Aug 01 00:32:31.088041 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:31.087944    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7902428de96a11c48a3cd2c538aca67dff9c949a109ec1e1865c69c729cff908"} err="failed to get container status \"7902428de96a11c48a3cd2c538aca67dff9c949a109ec1e1865c69c729cff908\": rpc error: code = NotFound desc = could not find container \"7902428de96a11c48a3cd2c538aca67dff9c949a109ec1e1865c69c729cff908\": container with ID starting with 7902428de96a11c48a3cd2c538aca67dff9c949a109ec1e1865c69c729cff908 not found: ID does not exist"
Aug 01 00:32:31.098988 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:31.098958    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"]
Aug 01 00:32:31.102349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:31.102320    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejpn78"]
Aug 01 00:32:31.947141 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:31.947110    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b8e62d3-1b92-497f-97d9-c5efa62536f6" path="/var/lib/kubelet/pods/7b8e62d3-1b92-497f-97d9-c5efa62536f6/volumes"
Aug 01 00:32:35.306561 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:35.306521    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:36.380256 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:36.380159    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:41.272458 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:41.272416    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:45.572177 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.572140    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:45.803003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.802961    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"]
Aug 01 00:32:45.803446 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.803429    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b8e62d3-1b92-497f-97d9-c5efa62536f6" containerName="main"
Aug 01 00:32:45.803506 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.803449    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b8e62d3-1b92-497f-97d9-c5efa62536f6" containerName="main"
Aug 01 00:32:45.803550 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.803540    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b8e62d3-1b92-497f-97d9-c5efa62536f6" containerName="main"
Aug 01 00:32:45.806727 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.806705    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"
Aug 01 00:32:45.809510 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.809450    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-dnj7v\""
Aug 01 00:32:45.809663 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.809525    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Aug 01 00:32:45.809663 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.809525    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-hntm9\""
Aug 01 00:32:45.809663 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.809527    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Aug 01 00:32:45.809663 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.809528    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Aug 01 00:32:45.813243 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.813222    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"]
Aug 01 00:32:45.826791 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.826698    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:45.898846 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.898803    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c805b31-8d5c-4084-bff5-5afc4f7eebd0-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9\" (UID: \"7c805b31-8d5c-4084-bff5-5afc4f7eebd0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"
Aug 01 00:32:45.898846 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.898850    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h8wg\" (UniqueName: \"kubernetes.io/projected/7c805b31-8d5c-4084-bff5-5afc4f7eebd0-kube-api-access-6h8wg\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9\" (UID: \"7c805b31-8d5c-4084-bff5-5afc4f7eebd0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"
Aug 01 00:32:45.999893 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.999854    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c805b31-8d5c-4084-bff5-5afc4f7eebd0-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9\" (UID: \"7c805b31-8d5c-4084-bff5-5afc4f7eebd0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"
Aug 01 00:32:46.000097 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:45.999906    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h8wg\" (UniqueName: \"kubernetes.io/projected/7c805b31-8d5c-4084-bff5-5afc4f7eebd0-kube-api-access-6h8wg\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9\" (UID: \"7c805b31-8d5c-4084-bff5-5afc4f7eebd0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"
Aug 01 00:32:46.002809 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:46.002775    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c805b31-8d5c-4084-bff5-5afc4f7eebd0-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9\" (UID: \"7c805b31-8d5c-4084-bff5-5afc4f7eebd0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"
Aug 01 00:32:46.009103 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:46.009068    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h8wg\" (UniqueName: \"kubernetes.io/projected/7c805b31-8d5c-4084-bff5-5afc4f7eebd0-kube-api-access-6h8wg\") pod \"custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9\" (UID: \"7c805b31-8d5c-4084-bff5-5afc4f7eebd0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"
Aug 01 00:32:46.119219 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:46.119121    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"
Aug 01 00:32:46.255164 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:46.255134    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"]
Aug 01 00:32:46.258012 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:32:46.257983    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c805b31_8d5c_4084_bff5_5afc4f7eebd0.slice/crio-6f027b7c9fdfba3b98946706536057a8be959507faecca94535337d083f615f5 WatchSource:0}: Error finding container 6f027b7c9fdfba3b98946706536057a8be959507faecca94535337d083f615f5: Status 404 returned error can't find the container with id 6f027b7c9fdfba3b98946706536057a8be959507faecca94535337d083f615f5
Aug 01 00:32:47.128047 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:47.128004    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:47.148718 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:47.148685    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9" event={"ID":"7c805b31-8d5c-4084-bff5-5afc4f7eebd0","Type":"ContainerStarted","Data":"e6990b57cdf231604365d59454a4063e11accce4d96447745f5fa7cd243a83b1"}
Aug 01 00:32:47.148901 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:47.148764    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9" event={"ID":"7c805b31-8d5c-4084-bff5-5afc4f7eebd0","Type":"ContainerStarted","Data":"6f027b7c9fdfba3b98946706536057a8be959507faecca94535337d083f615f5"}
Aug 01 00:32:47.148901 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:47.148828    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"
Aug 01 00:32:47.169349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:47.169290    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9" podStartSLOduration=2.169266147 podStartE2EDuration="2.169266147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:32:47.167542841 +0000 UTC m=+2809.861590258" watchObservedRunningTime="2026-08-01 00:32:47.169266147 +0000 UTC m=+2809.863313563"
Aug 01 00:32:47.221251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:47.221210    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:52.407503 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:32:52.407461    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:33:12.422633 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:33:12.422599    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:33:18.155356 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:33:18.155319    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"
Aug 01 00:34:00.132194 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:00.132116    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"]
Aug 01 00:34:00.140643 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:00.140615    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:00.239288 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:00.239254    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:00.900757 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:00.900717    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2" podUID="d5c022e2-6746-4845-9149-c87b72ea42bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:34:08.980093 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:08.980045    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:10.900124 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:10.900081    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2" podUID="d5c022e2-6746-4845-9149-c87b72ea42bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:34:15.151357 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.151290    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2" podUID="d5c022e2-6746-4845-9149-c87b72ea42bc" containerName="main" containerID="cri-o://bb195adf75c0a96f3affba9a7d0c06d26b295f11814ab1ef55614d35afed1538" gracePeriod=45
Aug 01 00:34:15.401027 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.401000    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"
Aug 01 00:34:15.457213 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.457140    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlq2c\" (UniqueName: \"kubernetes.io/projected/d5c022e2-6746-4845-9149-c87b72ea42bc-kube-api-access-xlq2c\") pod \"d5c022e2-6746-4845-9149-c87b72ea42bc\" (UID: \"d5c022e2-6746-4845-9149-c87b72ea42bc\") "
Aug 01 00:34:15.457213 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.457186    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c022e2-6746-4845-9149-c87b72ea42bc-tls-certs\") pod \"d5c022e2-6746-4845-9149-c87b72ea42bc\" (UID: \"d5c022e2-6746-4845-9149-c87b72ea42bc\") "
Aug 01 00:34:15.459297 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.459267    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c022e2-6746-4845-9149-c87b72ea42bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5c022e2-6746-4845-9149-c87b72ea42bc" (UID: "d5c022e2-6746-4845-9149-c87b72ea42bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:34:15.459411 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.459355    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c022e2-6746-4845-9149-c87b72ea42bc-kube-api-access-xlq2c" (OuterVolumeSpecName: "kube-api-access-xlq2c") pod "d5c022e2-6746-4845-9149-c87b72ea42bc" (UID: "d5c022e2-6746-4845-9149-c87b72ea42bc"). InnerVolumeSpecName "kube-api-access-xlq2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:34:15.527377 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.527342    2551 generic.go:356] "Generic (PLEG): container finished" podID="d5c022e2-6746-4845-9149-c87b72ea42bc" containerID="bb195adf75c0a96f3affba9a7d0c06d26b295f11814ab1ef55614d35afed1538" exitCode=0
Aug 01 00:34:15.527537 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.527387    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"
Aug 01 00:34:15.527537 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.527429    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2" event={"ID":"d5c022e2-6746-4845-9149-c87b72ea42bc","Type":"ContainerDied","Data":"bb195adf75c0a96f3affba9a7d0c06d26b295f11814ab1ef55614d35afed1538"}
Aug 01 00:34:15.527537 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.527482    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2" event={"ID":"d5c022e2-6746-4845-9149-c87b72ea42bc","Type":"ContainerDied","Data":"395bdde30ee958fd4f320b5d51d212f321e95cec1de1820094b845b1564e5bba"}
Aug 01 00:34:15.527537 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.527505    2551 scope.go:117] "RemoveContainer" containerID="bb195adf75c0a96f3affba9a7d0c06d26b295f11814ab1ef55614d35afed1538"
Aug 01 00:34:15.536457 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.536440    2551 scope.go:117] "RemoveContainer" containerID="bb195adf75c0a96f3affba9a7d0c06d26b295f11814ab1ef55614d35afed1538"
Aug 01 00:34:15.536728 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:34:15.536707    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb195adf75c0a96f3affba9a7d0c06d26b295f11814ab1ef55614d35afed1538\": container with ID starting with bb195adf75c0a96f3affba9a7d0c06d26b295f11814ab1ef55614d35afed1538 not found: ID does not exist" containerID="bb195adf75c0a96f3affba9a7d0c06d26b295f11814ab1ef55614d35afed1538"
Aug 01 00:34:15.536767 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.536737    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb195adf75c0a96f3affba9a7d0c06d26b295f11814ab1ef55614d35afed1538"} err="failed to get container status \"bb195adf75c0a96f3affba9a7d0c06d26b295f11814ab1ef55614d35afed1538\": rpc error: code = NotFound desc = could not find container \"bb195adf75c0a96f3affba9a7d0c06d26b295f11814ab1ef55614d35afed1538\": container with ID starting with bb195adf75c0a96f3affba9a7d0c06d26b295f11814ab1ef55614d35afed1538 not found: ID does not exist"
Aug 01 00:34:15.547069 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.547040    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"]
Aug 01 00:34:15.550666 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.550645    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-p7rg2"]
Aug 01 00:34:15.558452 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.558425    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5c022e2-6746-4845-9149-c87b72ea42bc-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:34:15.558452 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.558445    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlq2c\" (UniqueName: \"kubernetes.io/projected/d5c022e2-6746-4845-9149-c87b72ea42bc-kube-api-access-xlq2c\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:34:15.946468 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:15.946437    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c022e2-6746-4845-9149-c87b72ea42bc" path="/var/lib/kubelet/pods/d5c022e2-6746-4845-9149-c87b72ea42bc/volumes"
Aug 01 00:34:20.211488 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:20.211447    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:20.310829 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:20.310787    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:20.413374 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:20.413340    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:26.219274 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:26.219226    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:27.314963 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:27.314904    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:31.667465 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:31.667426    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:31.769247 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:31.769202    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:53.041466 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:53.041429    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"]
Aug 01 00:34:53.045603 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:53.045569    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:53.105802 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:53.105740    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:58.155424 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:34:58.155383    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9" podUID="7c805b31-8d5c-4084-bff5-5afc4f7eebd0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:35:08.062341 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.062299    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9" podUID="7c805b31-8d5c-4084-bff5-5afc4f7eebd0" containerName="main" containerID="cri-o://e6990b57cdf231604365d59454a4063e11accce4d96447745f5fa7cd243a83b1" gracePeriod=45
Aug 01 00:35:08.153624 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:35:08.153599    2551 logging.go:55] [core] [Channel #1033 SubChannel #1034]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.70:9003", ServerName: "10.133.0.70:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.70:9003: connect: connection refused"
Aug 01 00:35:08.305707 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.305678    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"
Aug 01 00:35:08.407771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.407668    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c805b31-8d5c-4084-bff5-5afc4f7eebd0-tls-certs\") pod \"7c805b31-8d5c-4084-bff5-5afc4f7eebd0\" (UID: \"7c805b31-8d5c-4084-bff5-5afc4f7eebd0\") "
Aug 01 00:35:08.407771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.407734    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h8wg\" (UniqueName: \"kubernetes.io/projected/7c805b31-8d5c-4084-bff5-5afc4f7eebd0-kube-api-access-6h8wg\") pod \"7c805b31-8d5c-4084-bff5-5afc4f7eebd0\" (UID: \"7c805b31-8d5c-4084-bff5-5afc4f7eebd0\") "
Aug 01 00:35:08.410049 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.410016    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c805b31-8d5c-4084-bff5-5afc4f7eebd0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c805b31-8d5c-4084-bff5-5afc4f7eebd0" (UID: "7c805b31-8d5c-4084-bff5-5afc4f7eebd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:35:08.410350 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.410315    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c805b31-8d5c-4084-bff5-5afc4f7eebd0-kube-api-access-6h8wg" (OuterVolumeSpecName: "kube-api-access-6h8wg") pod "7c805b31-8d5c-4084-bff5-5afc4f7eebd0" (UID: "7c805b31-8d5c-4084-bff5-5afc4f7eebd0"). InnerVolumeSpecName "kube-api-access-6h8wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:35:08.509166 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.509118    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c805b31-8d5c-4084-bff5-5afc4f7eebd0-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:35:08.509166 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.509167    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h8wg\" (UniqueName: \"kubernetes.io/projected/7c805b31-8d5c-4084-bff5-5afc4f7eebd0-kube-api-access-6h8wg\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:35:08.769617 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.769580    2551 generic.go:356] "Generic (PLEG): container finished" podID="7c805b31-8d5c-4084-bff5-5afc4f7eebd0" containerID="e6990b57cdf231604365d59454a4063e11accce4d96447745f5fa7cd243a83b1" exitCode=0
Aug 01 00:35:08.769787 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.769628    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"
Aug 01 00:35:08.769787 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.769660    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9" event={"ID":"7c805b31-8d5c-4084-bff5-5afc4f7eebd0","Type":"ContainerDied","Data":"e6990b57cdf231604365d59454a4063e11accce4d96447745f5fa7cd243a83b1"}
Aug 01 00:35:08.769787 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.769701    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9" event={"ID":"7c805b31-8d5c-4084-bff5-5afc4f7eebd0","Type":"ContainerDied","Data":"6f027b7c9fdfba3b98946706536057a8be959507faecca94535337d083f615f5"}
Aug 01 00:35:08.769787 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.769722    2551 scope.go:117] "RemoveContainer" containerID="e6990b57cdf231604365d59454a4063e11accce4d96447745f5fa7cd243a83b1"
Aug 01 00:35:08.778361 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.778345    2551 scope.go:117] "RemoveContainer" containerID="e6990b57cdf231604365d59454a4063e11accce4d96447745f5fa7cd243a83b1"
Aug 01 00:35:08.778604 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:35:08.778583    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6990b57cdf231604365d59454a4063e11accce4d96447745f5fa7cd243a83b1\": container with ID starting with e6990b57cdf231604365d59454a4063e11accce4d96447745f5fa7cd243a83b1 not found: ID does not exist" containerID="e6990b57cdf231604365d59454a4063e11accce4d96447745f5fa7cd243a83b1"
Aug 01 00:35:08.778650 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.778612    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6990b57cdf231604365d59454a4063e11accce4d96447745f5fa7cd243a83b1"} err="failed to get container status \"e6990b57cdf231604365d59454a4063e11accce4d96447745f5fa7cd243a83b1\": rpc error: code = NotFound desc = could not find container \"e6990b57cdf231604365d59454a4063e11accce4d96447745f5fa7cd243a83b1\": container with ID starting with e6990b57cdf231604365d59454a4063e11accce4d96447745f5fa7cd243a83b1 not found: ID does not exist"
Aug 01 00:35:08.789795 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.789758    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"]
Aug 01 00:35:08.791501 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:08.791482    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9"]
Aug 01 00:35:09.153620 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:09.153522    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-c9cb66892wgm9" podUID="7c805b31-8d5c-4084-bff5-5afc4f7eebd0" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.70:9003\" within 1s: context deadline exceeded"
Aug 01 00:35:09.946017 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:09.945984    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c805b31-8d5c-4084-bff5-5afc4f7eebd0" path="/var/lib/kubelet/pods/7c805b31-8d5c-4084-bff5-5afc4f7eebd0/volumes"
Aug 01 00:35:13.096383 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.096342    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:13.191086 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.191053    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:13.325978 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.325946    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"]
Aug 01 00:35:13.326318 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.326301    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c022e2-6746-4845-9149-c87b72ea42bc" containerName="main"
Aug 01 00:35:13.326402 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.326321    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c022e2-6746-4845-9149-c87b72ea42bc" containerName="main"
Aug 01 00:35:13.326402 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.326346    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c805b31-8d5c-4084-bff5-5afc4f7eebd0" containerName="main"
Aug 01 00:35:13.326402 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.326355    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c805b31-8d5c-4084-bff5-5afc4f7eebd0" containerName="main"
Aug 01 00:35:13.326556 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.326443    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c022e2-6746-4845-9149-c87b72ea42bc" containerName="main"
Aug 01 00:35:13.326556 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.326457    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c805b31-8d5c-4084-bff5-5afc4f7eebd0" containerName="main"
Aug 01 00:35:13.330746 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.330722    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.333551 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.333530    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-6tfw9\""
Aug 01 00:35:13.333669 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.333557    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Aug 01 00:35:13.333745 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.333664    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Aug 01 00:35:13.334153 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.334136    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Aug 01 00:35:13.343480 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.343404    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"]
Aug 01 00:35:13.351766 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.351689    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.351766 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.351744    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.351946 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.351773    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.351946 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.351819    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.351946 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.351867    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.351946 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.351894    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.352127 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.351950    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-844jk\" (UniqueName: \"kubernetes.io/projected/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-kube-api-access-844jk\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.352127 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.352042    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.352127 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.352073    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.357135 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.357112    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:13.448267 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.448232    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:13.453400 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.453374    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-844jk\" (UniqueName: \"kubernetes.io/projected/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-kube-api-access-844jk\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.453533 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.453434    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.453533 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.453456    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.453533 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.453475    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.453533 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.453494    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.453533 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.453509    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.453533 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.453532    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.453835 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.453619    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.453835 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.453659    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.453991 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.453970    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.454088 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.454067    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.454162 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.454116    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.454218 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.454172    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.454334 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.454313    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.455989 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.455965    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.456239 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.456216    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.460960 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.460938    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.461331 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.461312    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-844jk\" (UniqueName: \"kubernetes.io/projected/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-kube-api-access-844jk\") pod \"router-gateway-1-openshift-default-6cc7987576-8n8tz\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.644044 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.643944    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:13.776771 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.776738    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"]
Aug 01 00:35:13.779867 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:35:13.779834    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b80161b_b17b_4fc6_bf5a_6f33b03690c3.slice/crio-e2e722df9cc4e2139e8eb01c81c87c5ecb6bdce74a41510a83f695dd1acbb7fa WatchSource:0}: Error finding container e2e722df9cc4e2139e8eb01c81c87c5ecb6bdce74a41510a83f695dd1acbb7fa: Status 404 returned error can't find the container with id e2e722df9cc4e2139e8eb01c81c87c5ecb6bdce74a41510a83f695dd1acbb7fa
Aug 01 00:35:13.781901 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.781862    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:35:13.782022 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.781942    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:35:13.782022 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.781968    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:35:13.791421 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:13.791398    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz" event={"ID":"6b80161b-b17b-4fc6-bf5a-6f33b03690c3","Type":"ContainerStarted","Data":"e2e722df9cc4e2139e8eb01c81c87c5ecb6bdce74a41510a83f695dd1acbb7fa"}
Aug 01 00:35:14.797111 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:14.797073    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz" event={"ID":"6b80161b-b17b-4fc6-bf5a-6f33b03690c3","Type":"ContainerStarted","Data":"763df364675891c5a62c1d8023ba083167698625132ab3980cf1835954d6f01e"}
Aug 01 00:35:14.817209 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:14.817155    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz" podStartSLOduration=1.81714099 podStartE2EDuration="1.81714099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:35:14.815157447 +0000 UTC m=+2957.509204865" watchObservedRunningTime="2026-08-01 00:35:14.81714099 +0000 UTC m=+2957.511188406"
Aug 01 00:35:15.644201 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:15.644161    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:15.648842 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:15.648815    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:15.801986 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:15.801956    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:15.803021 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:15.802996    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:16.655730 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:16.655678    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:19.829645 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:19.829585    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:24.569977 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:24.569937    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:25.438511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:25.438474    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:25.536009 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:25.535973    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:27.866234 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:27.866201    2551 generic.go:356] "Generic (PLEG): container finished" podID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerID="763df364675891c5a62c1d8023ba083167698625132ab3980cf1835954d6f01e" exitCode=0
Aug 01 00:35:27.866668 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:27.866286    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz" event={"ID":"6b80161b-b17b-4fc6-bf5a-6f33b03690c3","Type":"ContainerDied","Data":"763df364675891c5a62c1d8023ba083167698625132ab3980cf1835954d6f01e"}
Aug 01 00:35:27.866711 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:27.866696    2551 scope.go:117] "RemoveContainer" containerID="763df364675891c5a62c1d8023ba083167698625132ab3980cf1835954d6f01e"
Aug 01 00:35:27.867605 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:27.867577    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:35:27.867712 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:27.867640    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:35:27.867712 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:27.867672    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:35:28.873855 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:28.873818    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz" event={"ID":"6b80161b-b17b-4fc6-bf5a-6f33b03690c3","Type":"ContainerStarted","Data":"a5c34ec6c10c37ceffb89c4059c69c1de52aa866a4d0111044bdb92e0aee674a"}
Aug 01 00:35:29.644983 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:29.644939    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:29.646264 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:29.646231    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz" podUID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Aug 01 00:35:30.645050 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:30.645007    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz" podUID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Aug 01 00:35:30.802548 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:30.802504    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:31.645153 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:31.645098    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz" podUID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Aug 01 00:35:32.644436 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:32.644388    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz" podUID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Aug 01 00:35:33.644817 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:33.644769    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz" podUID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Aug 01 00:35:34.648903 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:34.648868    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:34.649826 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:34.649806    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:35:37.623843 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:37.623806    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:37.833108 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:37.833075    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"]
Aug 01 00:35:37.835495 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:37.835474    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"
Aug 01 00:35:37.838030 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:37.838002    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-96ljt\""
Aug 01 00:35:37.838030 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:37.838019    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-b4k48\""
Aug 01 00:35:37.838226 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:37.838149    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Aug 01 00:35:37.845860 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:37.845836    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"]
Aug 01 00:35:37.864022 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:37.863997    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:37.997995 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:37.997966    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66271719-8333-4973-bd7a-782398dbe4a6-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt\" (UID: \"66271719-8333-4973-bd7a-782398dbe4a6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"
Aug 01 00:35:37.998162 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:37.998086    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwdtk\" (UniqueName: \"kubernetes.io/projected/66271719-8333-4973-bd7a-782398dbe4a6-kube-api-access-zwdtk\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt\" (UID: \"66271719-8333-4973-bd7a-782398dbe4a6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"
Aug 01 00:35:38.063006 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:38.062914    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:38.098829 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:38.098793    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwdtk\" (UniqueName: \"kubernetes.io/projected/66271719-8333-4973-bd7a-782398dbe4a6-kube-api-access-zwdtk\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt\" (UID: \"66271719-8333-4973-bd7a-782398dbe4a6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"
Aug 01 00:35:38.099054 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:38.098860    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66271719-8333-4973-bd7a-782398dbe4a6-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt\" (UID: \"66271719-8333-4973-bd7a-782398dbe4a6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"
Aug 01 00:35:38.101742 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:38.101695    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66271719-8333-4973-bd7a-782398dbe4a6-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt\" (UID: \"66271719-8333-4973-bd7a-782398dbe4a6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"
Aug 01 00:35:38.107402 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:38.107367    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwdtk\" (UniqueName: \"kubernetes.io/projected/66271719-8333-4973-bd7a-782398dbe4a6-kube-api-access-zwdtk\") pod \"router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt\" (UID: \"66271719-8333-4973-bd7a-782398dbe4a6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"
Aug 01 00:35:38.147497 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:38.147462    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"
Aug 01 00:35:38.302472 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:38.302386    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"]
Aug 01 00:35:38.305566 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:35:38.305539    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66271719_8333_4973_bd7a_782398dbe4a6.slice/crio-06af71ac15ef694d22a0127cc76035212c8d9fcd10f8820f94163d33b31ee686 WatchSource:0}: Error finding container 06af71ac15ef694d22a0127cc76035212c8d9fcd10f8820f94163d33b31ee686: Status 404 returned error can't find the container with id 06af71ac15ef694d22a0127cc76035212c8d9fcd10f8820f94163d33b31ee686
Aug 01 00:35:38.932508 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:38.932466    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt" event={"ID":"66271719-8333-4973-bd7a-782398dbe4a6","Type":"ContainerStarted","Data":"0ef60f5698c3381e746facc13695a957e6f59a47e6fcd171487471f6436549a4"}
Aug 01 00:35:38.933031 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:38.932527    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt" event={"ID":"66271719-8333-4973-bd7a-782398dbe4a6","Type":"ContainerStarted","Data":"06af71ac15ef694d22a0127cc76035212c8d9fcd10f8820f94163d33b31ee686"}
Aug 01 00:35:38.933031 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:38.932614    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"
Aug 01 00:35:38.948198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:35:38.948156    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt" podStartSLOduration=1.948143816 podStartE2EDuration="1.948143816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:35:38.946160849 +0000 UTC m=+2981.640208265" watchObservedRunningTime="2026-08-01 00:35:38.948143816 +0000 UTC m=+2981.642191231"
Aug 01 00:36:00.856464 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:36:00.856419    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:36:00.908236 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:36:00.908178    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:36:09.938168 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:36:09.938134    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"
Aug 01 00:36:20.378443 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:36:20.378387    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:36:40.448213 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:36:40.448172    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:36:40.550788 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:36:40.550752    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:36:40.650882 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:36:40.650845    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:36:46.159099 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:36:46.159067    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:36:46.220001 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:36:46.219954    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:36:54.451794 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:36:54.451688    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:37:42.433566 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:42.433528    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:37:42.438847 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:42.438817    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"]
Aug 01 00:37:43.741290 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:43.741248    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:37:49.938079 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:49.938042    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt" podUID="66271719-8333-4973-bd7a-782398dbe4a6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:37:57.458057 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:57.458013    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt" podUID="66271719-8333-4973-bd7a-782398dbe4a6" containerName="main" containerID="cri-o://0ef60f5698c3381e746facc13695a957e6f59a47e6fcd171487471f6436549a4" gracePeriod=45
Aug 01 00:37:57.591600 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:57.591569    2551 generic.go:356] "Generic (PLEG): container finished" podID="66271719-8333-4973-bd7a-782398dbe4a6" containerID="0ef60f5698c3381e746facc13695a957e6f59a47e6fcd171487471f6436549a4" exitCode=0
Aug 01 00:37:57.591749 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:57.591607    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt" event={"ID":"66271719-8333-4973-bd7a-782398dbe4a6","Type":"ContainerDied","Data":"0ef60f5698c3381e746facc13695a957e6f59a47e6fcd171487471f6436549a4"}
Aug 01 00:37:57.706205 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:57.706179    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"
Aug 01 00:37:57.767376 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:57.767344    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwdtk\" (UniqueName: \"kubernetes.io/projected/66271719-8333-4973-bd7a-782398dbe4a6-kube-api-access-zwdtk\") pod \"66271719-8333-4973-bd7a-782398dbe4a6\" (UID: \"66271719-8333-4973-bd7a-782398dbe4a6\") "
Aug 01 00:37:57.767585 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:57.767438    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66271719-8333-4973-bd7a-782398dbe4a6-tls-certs\") pod \"66271719-8333-4973-bd7a-782398dbe4a6\" (UID: \"66271719-8333-4973-bd7a-782398dbe4a6\") "
Aug 01 00:37:57.769521 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:57.769496    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66271719-8333-4973-bd7a-782398dbe4a6-kube-api-access-zwdtk" (OuterVolumeSpecName: "kube-api-access-zwdtk") pod "66271719-8333-4973-bd7a-782398dbe4a6" (UID: "66271719-8333-4973-bd7a-782398dbe4a6"). InnerVolumeSpecName "kube-api-access-zwdtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:37:57.769613 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:57.769521    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66271719-8333-4973-bd7a-782398dbe4a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66271719-8333-4973-bd7a-782398dbe4a6" (UID: "66271719-8333-4973-bd7a-782398dbe4a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:37:57.868304 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:57.868263    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66271719-8333-4973-bd7a-782398dbe4a6-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:37:57.868304 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:57.868302    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwdtk\" (UniqueName: \"kubernetes.io/projected/66271719-8333-4973-bd7a-782398dbe4a6-kube-api-access-zwdtk\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:37:58.597296 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:58.597256    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt" event={"ID":"66271719-8333-4973-bd7a-782398dbe4a6","Type":"ContainerDied","Data":"06af71ac15ef694d22a0127cc76035212c8d9fcd10f8820f94163d33b31ee686"}
Aug 01 00:37:58.597797 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:58.597302    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"
Aug 01 00:37:58.597797 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:58.597322    2551 scope.go:117] "RemoveContainer" containerID="0ef60f5698c3381e746facc13695a957e6f59a47e6fcd171487471f6436549a4"
Aug 01 00:37:58.611992 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:58.611960    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"]
Aug 01 00:37:58.619833 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:58.619803    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-6d9fd8bcc-vx9kt"]
Aug 01 00:37:59.947183 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:37:59.947150    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66271719-8333-4973-bd7a-782398dbe4a6" path="/var/lib/kubelet/pods/66271719-8333-4973-bd7a-782398dbe4a6/volumes"
Aug 01 00:38:02.408692 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:02.408651    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:02.511517 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:02.511468    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:07.354346 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:07.354309    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"]
Aug 01 00:38:07.354767 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:07.354515    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz" podUID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerName="istio-proxy" containerID="cri-o://a5c34ec6c10c37ceffb89c4059c69c1de52aa866a4d0111044bdb92e0aee674a" gracePeriod=30
Aug 01 00:38:07.396165 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:07.396120    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:12.597089 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.597060    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:38:12.662768 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.662737    2551 generic.go:356] "Generic (PLEG): container finished" podID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerID="a5c34ec6c10c37ceffb89c4059c69c1de52aa866a4d0111044bdb92e0aee674a" exitCode=0
Aug 01 00:38:12.662910 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.662799    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"
Aug 01 00:38:12.662910 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.662822    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz" event={"ID":"6b80161b-b17b-4fc6-bf5a-6f33b03690c3","Type":"ContainerDied","Data":"a5c34ec6c10c37ceffb89c4059c69c1de52aa866a4d0111044bdb92e0aee674a"}
Aug 01 00:38:12.662910 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.662898    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz" event={"ID":"6b80161b-b17b-4fc6-bf5a-6f33b03690c3","Type":"ContainerDied","Data":"e2e722df9cc4e2139e8eb01c81c87c5ecb6bdce74a41510a83f695dd1acbb7fa"}
Aug 01 00:38:12.663122 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.663006    2551 scope.go:117] "RemoveContainer" containerID="a5c34ec6c10c37ceffb89c4059c69c1de52aa866a4d0111044bdb92e0aee674a"
Aug 01 00:38:12.672827 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.672808    2551 scope.go:117] "RemoveContainer" containerID="763df364675891c5a62c1d8023ba083167698625132ab3980cf1835954d6f01e"
Aug 01 00:38:12.680323 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.680299    2551 scope.go:117] "RemoveContainer" containerID="a5c34ec6c10c37ceffb89c4059c69c1de52aa866a4d0111044bdb92e0aee674a"
Aug 01 00:38:12.680579 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:38:12.680561    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c34ec6c10c37ceffb89c4059c69c1de52aa866a4d0111044bdb92e0aee674a\": container with ID starting with a5c34ec6c10c37ceffb89c4059c69c1de52aa866a4d0111044bdb92e0aee674a not found: ID does not exist" containerID="a5c34ec6c10c37ceffb89c4059c69c1de52aa866a4d0111044bdb92e0aee674a"
Aug 01 00:38:12.680637 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.680591    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c34ec6c10c37ceffb89c4059c69c1de52aa866a4d0111044bdb92e0aee674a"} err="failed to get container status \"a5c34ec6c10c37ceffb89c4059c69c1de52aa866a4d0111044bdb92e0aee674a\": rpc error: code = NotFound desc = could not find container \"a5c34ec6c10c37ceffb89c4059c69c1de52aa866a4d0111044bdb92e0aee674a\": container with ID starting with a5c34ec6c10c37ceffb89c4059c69c1de52aa866a4d0111044bdb92e0aee674a not found: ID does not exist"
Aug 01 00:38:12.680637 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.680609    2551 scope.go:117] "RemoveContainer" containerID="763df364675891c5a62c1d8023ba083167698625132ab3980cf1835954d6f01e"
Aug 01 00:38:12.680867 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:38:12.680846    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763df364675891c5a62c1d8023ba083167698625132ab3980cf1835954d6f01e\": container with ID starting with 763df364675891c5a62c1d8023ba083167698625132ab3980cf1835954d6f01e not found: ID does not exist" containerID="763df364675891c5a62c1d8023ba083167698625132ab3980cf1835954d6f01e"
Aug 01 00:38:12.680964 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.680877    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763df364675891c5a62c1d8023ba083167698625132ab3980cf1835954d6f01e"} err="failed to get container status \"763df364675891c5a62c1d8023ba083167698625132ab3980cf1835954d6f01e\": rpc error: code = NotFound desc = could not find container \"763df364675891c5a62c1d8023ba083167698625132ab3980cf1835954d6f01e\": container with ID starting with 763df364675891c5a62c1d8023ba083167698625132ab3980cf1835954d6f01e not found: ID does not exist"
Aug 01 00:38:12.693156 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693133    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-data\") pod \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") "
Aug 01 00:38:12.693237 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693202    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-workload-certs\") pod \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") "
Aug 01 00:38:12.693237 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693220    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-envoy\") pod \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") "
Aug 01 00:38:12.693352 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693237    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-844jk\" (UniqueName: \"kubernetes.io/projected/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-kube-api-access-844jk\") pod \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") "
Aug 01 00:38:12.693352 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693261    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-workload-socket\") pod \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") "
Aug 01 00:38:12.693352 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693280    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-credential-socket\") pod \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") "
Aug 01 00:38:12.693352 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693324    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b80161b-b17b-4fc6-bf5a-6f33b03690c3" (UID: "6b80161b-b17b-4fc6-bf5a-6f33b03690c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:38:12.693352 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693336    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istiod-ca-cert\") pod \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") "
Aug 01 00:38:12.693606 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693365    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-podinfo\") pod \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") "
Aug 01 00:38:12.693606 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693393    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-token\") pod \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\" (UID: \"6b80161b-b17b-4fc6-bf5a-6f33b03690c3\") "
Aug 01 00:38:12.693723 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693604    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:38:12.693723 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693660    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b80161b-b17b-4fc6-bf5a-6f33b03690c3" (UID: "6b80161b-b17b-4fc6-bf5a-6f33b03690c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:38:12.693723 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693676    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b80161b-b17b-4fc6-bf5a-6f33b03690c3" (UID: "6b80161b-b17b-4fc6-bf5a-6f33b03690c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:38:12.693938 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.693892    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b80161b-b17b-4fc6-bf5a-6f33b03690c3" (UID: "6b80161b-b17b-4fc6-bf5a-6f33b03690c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:38:12.694155 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.694127    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b80161b-b17b-4fc6-bf5a-6f33b03690c3" (UID: "6b80161b-b17b-4fc6-bf5a-6f33b03690c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:38:12.695508 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.695481    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-kube-api-access-844jk" (OuterVolumeSpecName: "kube-api-access-844jk") pod "6b80161b-b17b-4fc6-bf5a-6f33b03690c3" (UID: "6b80161b-b17b-4fc6-bf5a-6f33b03690c3"). InnerVolumeSpecName "kube-api-access-844jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:38:12.695592 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.695514    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b80161b-b17b-4fc6-bf5a-6f33b03690c3" (UID: "6b80161b-b17b-4fc6-bf5a-6f33b03690c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:38:12.695913 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.695898    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b80161b-b17b-4fc6-bf5a-6f33b03690c3" (UID: "6b80161b-b17b-4fc6-bf5a-6f33b03690c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:38:12.696000 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.695948    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b80161b-b17b-4fc6-bf5a-6f33b03690c3" (UID: "6b80161b-b17b-4fc6-bf5a-6f33b03690c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 00:38:12.794370 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.794327    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:38:12.794370 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.794359    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:38:12.794370 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.794368    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:38:12.794370 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.794377    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:38:12.794671 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.794385    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:38:12.794671 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.794394    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-844jk\" (UniqueName: \"kubernetes.io/projected/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-kube-api-access-844jk\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:38:12.794671 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.794405    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:38:12.794671 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.794414    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b80161b-b17b-4fc6-bf5a-6f33b03690c3-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:38:12.985130 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.985101    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"]
Aug 01 00:38:12.988362 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:12.988333    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-8n8tz"]
Aug 01 00:38:13.949452 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:13.949415    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" path="/var/lib/kubelet/pods/6b80161b-b17b-4fc6-bf5a-6f33b03690c3/volumes"
Aug 01 00:38:17.007756 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.007722    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"]
Aug 01 00:38:17.008256 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.008167    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerName="istio-proxy"
Aug 01 00:38:17.008256 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.008185    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerName="istio-proxy"
Aug 01 00:38:17.008256 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.008196    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66271719-8333-4973-bd7a-782398dbe4a6" containerName="main"
Aug 01 00:38:17.008256 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.008204    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="66271719-8333-4973-bd7a-782398dbe4a6" containerName="main"
Aug 01 00:38:17.008479 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.008303    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerName="istio-proxy"
Aug 01 00:38:17.008479 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.008316    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="66271719-8333-4973-bd7a-782398dbe4a6" containerName="main"
Aug 01 00:38:17.008479 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.008332    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerName="istio-proxy"
Aug 01 00:38:17.008479 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.008420    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerName="istio-proxy"
Aug 01 00:38:17.008479 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.008430    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b80161b-b17b-4fc6-bf5a-6f33b03690c3" containerName="istio-proxy"
Aug 01 00:38:17.012979 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.012956    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.015008 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.014983    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-tzdbw\""
Aug 01 00:38:17.015158 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.015122    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Aug 01 00:38:17.015388 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.015371    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Aug 01 00:38:17.015728 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.015707    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Aug 01 00:38:17.015821 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.015786    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:38:17.015963 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.015869    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-mgzxr\""
Aug 01 00:38:17.015963 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.015952    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Aug 01 00:38:17.027755 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.027730    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"]
Aug 01 00:38:17.080466 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.080430    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:17.142816 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.142780    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.142816 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.142837    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbgq\" (UniqueName: \"kubernetes.io/projected/0155c08f-058c-470c-af80-7f79013365a3-kube-api-access-xvbgq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.143134 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.142868    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.143134 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.142891    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0155c08f-058c-470c-af80-7f79013365a3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.143134 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.142946    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.143134 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.143031    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.143134 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.143071    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0155c08f-058c-470c-af80-7f79013365a3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.143311 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.143164    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.244697 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.244654    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.244880 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.244703    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.244880 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.244755    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbgq\" (UniqueName: \"kubernetes.io/projected/0155c08f-058c-470c-af80-7f79013365a3-kube-api-access-xvbgq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.244880 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.244783    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.244880 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.244820    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0155c08f-058c-470c-af80-7f79013365a3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.244880 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.244861    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.245198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.244898    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.245198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.244942    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0155c08f-058c-470c-af80-7f79013365a3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.245198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.245115    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.245877 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.245530    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.245877 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.245760    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0155c08f-058c-470c-af80-7f79013365a3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.245877 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.245831    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.246392 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.246350    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.247758 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.247729    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.247957 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.247913    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0155c08f-058c-470c-af80-7f79013365a3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.248118 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.248095    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"]
Aug 01 00:38:17.256754 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.256558    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"
Aug 01 00:38:17.259839 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.259782    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-jmt7t\""
Aug 01 00:38:17.262713 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.262662    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbgq\" (UniqueName: \"kubernetes.io/projected/0155c08f-058c-470c-af80-7f79013365a3-kube-api-access-xvbgq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.262852 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.262828    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"]
Aug 01 00:38:17.282761 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.282726    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:17.322397 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.322373    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:17.345610 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.345581    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j5cn\" (UniqueName: \"kubernetes.io/projected/953ac4e8-ba98-47f4-ac83-281cf2c53e5b-kube-api-access-2j5cn\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9\" (UID: \"953ac4e8-ba98-47f4-ac83-281cf2c53e5b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"
Aug 01 00:38:17.345737 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.345627    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953ac4e8-ba98-47f4-ac83-281cf2c53e5b-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9\" (UID: \"953ac4e8-ba98-47f4-ac83-281cf2c53e5b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"
Aug 01 00:38:17.446420 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.446386    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j5cn\" (UniqueName: \"kubernetes.io/projected/953ac4e8-ba98-47f4-ac83-281cf2c53e5b-kube-api-access-2j5cn\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9\" (UID: \"953ac4e8-ba98-47f4-ac83-281cf2c53e5b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"
Aug 01 00:38:17.446603 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.446443    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953ac4e8-ba98-47f4-ac83-281cf2c53e5b-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9\" (UID: \"953ac4e8-ba98-47f4-ac83-281cf2c53e5b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"
Aug 01 00:38:17.449555 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.449457    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953ac4e8-ba98-47f4-ac83-281cf2c53e5b-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9\" (UID: \"953ac4e8-ba98-47f4-ac83-281cf2c53e5b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"
Aug 01 00:38:17.454063 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.454040    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j5cn\" (UniqueName: \"kubernetes.io/projected/953ac4e8-ba98-47f4-ac83-281cf2c53e5b-kube-api-access-2j5cn\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9\" (UID: \"953ac4e8-ba98-47f4-ac83-281cf2c53e5b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"
Aug 01 00:38:17.470239 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.470220    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"]
Aug 01 00:38:17.473487 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:38:17.473460    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0155c08f_058c_470c_af80_7f79013365a3.slice/crio-5f05b2336d78b58bf4805e3dfa90c05ed5963db4ffd3a29bd3a3d449263db067 WatchSource:0}: Error finding container 5f05b2336d78b58bf4805e3dfa90c05ed5963db4ffd3a29bd3a3d449263db067: Status 404 returned error can't find the container with id 5f05b2336d78b58bf4805e3dfa90c05ed5963db4ffd3a29bd3a3d449263db067
Aug 01 00:38:17.476173 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.476148    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:38:17.581646 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.581534    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"
Aug 01 00:38:17.687775 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.687736    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" event={"ID":"0155c08f-058c-470c-af80-7f79013365a3","Type":"ContainerStarted","Data":"5f05b2336d78b58bf4805e3dfa90c05ed5963db4ffd3a29bd3a3d449263db067"}
Aug 01 00:38:17.733346 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:17.733311    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"]
Aug 01 00:38:17.736306 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:38:17.736266    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953ac4e8_ba98_47f4_ac83_281cf2c53e5b.slice/crio-a564ced4a4fe3831dcbd9faa15f38c169d3771e77a59cae0e1a5a1a751b06900 WatchSource:0}: Error finding container a564ced4a4fe3831dcbd9faa15f38c169d3771e77a59cae0e1a5a1a751b06900: Status 404 returned error can't find the container with id a564ced4a4fe3831dcbd9faa15f38c169d3771e77a59cae0e1a5a1a751b06900
Aug 01 00:38:18.695459 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:18.695417    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9" event={"ID":"953ac4e8-ba98-47f4-ac83-281cf2c53e5b","Type":"ContainerStarted","Data":"050dd3ac0b3ec4187978f1eeb9f8f1c8bbe5c370794055cc663b8171a102d831"}
Aug 01 00:38:18.695982 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:18.695481    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9" event={"ID":"953ac4e8-ba98-47f4-ac83-281cf2c53e5b","Type":"ContainerStarted","Data":"a564ced4a4fe3831dcbd9faa15f38c169d3771e77a59cae0e1a5a1a751b06900"}
Aug 01 00:38:18.695982 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:18.695692    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"
Aug 01 00:38:18.712544 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:18.711900    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9" podStartSLOduration=1.7118779750000002 podStartE2EDuration="1.711877975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:18.709846843 +0000 UTC m=+3141.403894260" watchObservedRunningTime="2026-08-01 00:38:18.711877975 +0000 UTC m=+3141.405925393"
Aug 01 00:38:19.701184 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:19.701146    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" event={"ID":"0155c08f-058c-470c-af80-7f79013365a3","Type":"ContainerStarted","Data":"f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166"}
Aug 01 00:38:19.701677 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:19.701252    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:20.709396 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:20.709295    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" event={"ID":"0155c08f-058c-470c-af80-7f79013365a3","Type":"ContainerStarted","Data":"7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07"}
Aug 01 00:38:22.721126 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:22.721092    2551 generic.go:356] "Generic (PLEG): container finished" podID="0155c08f-058c-470c-af80-7f79013365a3" containerID="7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07" exitCode=0
Aug 01 00:38:22.721518 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:22.721142    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" event={"ID":"0155c08f-058c-470c-af80-7f79013365a3","Type":"ContainerDied","Data":"7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07"}
Aug 01 00:38:23.732463 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:23.732411    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" event={"ID":"0155c08f-058c-470c-af80-7f79013365a3","Type":"ContainerStarted","Data":"03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608"}
Aug 01 00:38:23.756894 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:23.756824    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podStartSLOduration=4.206846353 podStartE2EDuration="7.756799823s" totalImagesPullingTime="1.326253041s" totalInitContainerRuntime="2.223700429s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:38:23.753885555 +0000 UTC m=+3146.447933011" watchObservedRunningTime="2026-08-01 00:38:23.756799823 +0000 UTC m=+3146.450847240"
Aug 01 00:38:27.322783 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:27.322741    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:27.323208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:27.322844    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:27.324206 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:27.324177    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Aug 01 00:38:28.998387 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:28.998333    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:30.373518 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:30.373480    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:35.819448 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:35.819398    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:37.323368 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:37.323318    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Aug 01 00:38:37.343497 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:37.343459    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:38:37.661450 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:37.661359    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:44.297049 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:44.297003    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:44.455685 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:44.455647    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:47.323120 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:47.323070    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Aug 01 00:38:49.704017 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:49.703977    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"
Aug 01 00:38:50.678956 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:50.678900    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:54.678315 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:54.678279    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:57.323742 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:57.323696    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Aug 01 00:38:58.530308 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:58.530203    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:58.595288 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:38:58.595228    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:07.323002 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:07.322948    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Aug 01 00:39:14.618693 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:14.618653    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:14.716857 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:14.716818    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:14.816720 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:14.816669    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:17.323238 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:17.323168    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Aug 01 00:39:20.516690 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:20.516632    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:20.580485 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:20.580429    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:22.180671 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:22.180633    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:22.437314 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:22.437188    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:27.323554 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:27.323489    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Aug 01 00:39:35.020127 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:35.020074    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:37.323294 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:37.323234    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Aug 01 00:39:47.323252 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:47.323203    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Aug 01 00:39:57.323093 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:39:57.322980    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Aug 01 00:40:07.323084 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:07.323034    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Aug 01 00:40:17.332702 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:17.332669    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:40:17.345498 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:17.345471    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:40:26.096562 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:26.096515    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:40:26.101241 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:26.101204    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"]
Aug 01 00:40:26.112056 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:26.112026    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"]
Aug 01 00:40:26.160345 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:26.160309    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:40:29.703481 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:29.703430    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9" podUID="953ac4e8-ba98-47f4-ac83-281cf2c53e5b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:40:39.702982 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:39.702940    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9" podUID="953ac4e8-ba98-47f4-ac83-281cf2c53e5b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:40:41.123766 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.123702    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9" podUID="953ac4e8-ba98-47f4-ac83-281cf2c53e5b" containerName="main" containerID="cri-o://050dd3ac0b3ec4187978f1eeb9f8f1c8bbe5c370794055cc663b8171a102d831" gracePeriod=45
Aug 01 00:40:41.140372 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.140340    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main" containerID="cri-o://03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608" gracePeriod=45
Aug 01 00:40:41.363970 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.363938    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"
Aug 01 00:40:41.415022 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.414909    2551 generic.go:356] "Generic (PLEG): container finished" podID="953ac4e8-ba98-47f4-ac83-281cf2c53e5b" containerID="050dd3ac0b3ec4187978f1eeb9f8f1c8bbe5c370794055cc663b8171a102d831" exitCode=0
Aug 01 00:40:41.415022 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.414973    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"
Aug 01 00:40:41.415022 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.414993    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9" event={"ID":"953ac4e8-ba98-47f4-ac83-281cf2c53e5b","Type":"ContainerDied","Data":"050dd3ac0b3ec4187978f1eeb9f8f1c8bbe5c370794055cc663b8171a102d831"}
Aug 01 00:40:41.415299 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.415060    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9" event={"ID":"953ac4e8-ba98-47f4-ac83-281cf2c53e5b","Type":"ContainerDied","Data":"a564ced4a4fe3831dcbd9faa15f38c169d3771e77a59cae0e1a5a1a751b06900"}
Aug 01 00:40:41.415299 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.415091    2551 scope.go:117] "RemoveContainer" containerID="050dd3ac0b3ec4187978f1eeb9f8f1c8bbe5c370794055cc663b8171a102d831"
Aug 01 00:40:41.424603 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.424579    2551 scope.go:117] "RemoveContainer" containerID="050dd3ac0b3ec4187978f1eeb9f8f1c8bbe5c370794055cc663b8171a102d831"
Aug 01 00:40:41.424890 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:40:41.424854    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050dd3ac0b3ec4187978f1eeb9f8f1c8bbe5c370794055cc663b8171a102d831\": container with ID starting with 050dd3ac0b3ec4187978f1eeb9f8f1c8bbe5c370794055cc663b8171a102d831 not found: ID does not exist" containerID="050dd3ac0b3ec4187978f1eeb9f8f1c8bbe5c370794055cc663b8171a102d831"
Aug 01 00:40:41.424981 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.424899    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050dd3ac0b3ec4187978f1eeb9f8f1c8bbe5c370794055cc663b8171a102d831"} err="failed to get container status \"050dd3ac0b3ec4187978f1eeb9f8f1c8bbe5c370794055cc663b8171a102d831\": rpc error: code = NotFound desc = could not find container \"050dd3ac0b3ec4187978f1eeb9f8f1c8bbe5c370794055cc663b8171a102d831\": container with ID starting with 050dd3ac0b3ec4187978f1eeb9f8f1c8bbe5c370794055cc663b8171a102d831 not found: ID does not exist"
Aug 01 00:40:41.454785 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.454761    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953ac4e8-ba98-47f4-ac83-281cf2c53e5b-tls-certs\") pod \"953ac4e8-ba98-47f4-ac83-281cf2c53e5b\" (UID: \"953ac4e8-ba98-47f4-ac83-281cf2c53e5b\") "
Aug 01 00:40:41.454946 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.454801    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j5cn\" (UniqueName: \"kubernetes.io/projected/953ac4e8-ba98-47f4-ac83-281cf2c53e5b-kube-api-access-2j5cn\") pod \"953ac4e8-ba98-47f4-ac83-281cf2c53e5b\" (UID: \"953ac4e8-ba98-47f4-ac83-281cf2c53e5b\") "
Aug 01 00:40:41.456990 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.456967    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953ac4e8-ba98-47f4-ac83-281cf2c53e5b-kube-api-access-2j5cn" (OuterVolumeSpecName: "kube-api-access-2j5cn") pod "953ac4e8-ba98-47f4-ac83-281cf2c53e5b" (UID: "953ac4e8-ba98-47f4-ac83-281cf2c53e5b"). InnerVolumeSpecName "kube-api-access-2j5cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:40:41.457126 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.457110    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/953ac4e8-ba98-47f4-ac83-281cf2c53e5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "953ac4e8-ba98-47f4-ac83-281cf2c53e5b" (UID: "953ac4e8-ba98-47f4-ac83-281cf2c53e5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:40:41.555734 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.555697    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953ac4e8-ba98-47f4-ac83-281cf2c53e5b-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:40:41.555734 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.555731    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j5cn\" (UniqueName: \"kubernetes.io/projected/953ac4e8-ba98-47f4-ac83-281cf2c53e5b-kube-api-access-2j5cn\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:40:41.735463 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.735431    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"]
Aug 01 00:40:41.738854 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.738829    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezv6s9"]
Aug 01 00:40:41.755321 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.755265    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="llm-d-routing-sidecar" containerID="cri-o://f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166" gracePeriod=45
Aug 01 00:40:41.948110 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.948073    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="953ac4e8-ba98-47f4-ac83-281cf2c53e5b" path="/var/lib/kubelet/pods/953ac4e8-ba98-47f4-ac83-281cf2c53e5b/volumes"
Aug 01 00:40:41.986555 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:41.986529    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:40:42.059953 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.059893    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvbgq\" (UniqueName: \"kubernetes.io/projected/0155c08f-058c-470c-af80-7f79013365a3-kube-api-access-xvbgq\") pod \"0155c08f-058c-470c-af80-7f79013365a3\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") "
Aug 01 00:40:42.059953 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.059955    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-model-cache\") pod \"0155c08f-058c-470c-af80-7f79013365a3\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") "
Aug 01 00:40:42.060198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.059990    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0155c08f-058c-470c-af80-7f79013365a3-cabundle-cert\") pod \"0155c08f-058c-470c-af80-7f79013365a3\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") "
Aug 01 00:40:42.060198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.060026    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-tmp-dir\") pod \"0155c08f-058c-470c-af80-7f79013365a3\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") "
Aug 01 00:40:42.060198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.060042    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0155c08f-058c-470c-af80-7f79013365a3-tls-certs\") pod \"0155c08f-058c-470c-af80-7f79013365a3\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") "
Aug 01 00:40:42.060198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.060062    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-dshm\") pod \"0155c08f-058c-470c-af80-7f79013365a3\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") "
Aug 01 00:40:42.060198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.060088    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-home\") pod \"0155c08f-058c-470c-af80-7f79013365a3\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") "
Aug 01 00:40:42.060198 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.060160    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-kserve-provision-location\") pod \"0155c08f-058c-470c-af80-7f79013365a3\" (UID: \"0155c08f-058c-470c-af80-7f79013365a3\") "
Aug 01 00:40:42.060515 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.060324    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "0155c08f-058c-470c-af80-7f79013365a3" (UID: "0155c08f-058c-470c-af80-7f79013365a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:40:42.060515 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.060411    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0155c08f-058c-470c-af80-7f79013365a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0155c08f-058c-470c-af80-7f79013365a3" (UID: "0155c08f-058c-470c-af80-7f79013365a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:40:42.060624 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.060527    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.060884 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.060856    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-home" (OuterVolumeSpecName: "home") pod "0155c08f-058c-470c-af80-7f79013365a3" (UID: "0155c08f-058c-470c-af80-7f79013365a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:40:42.062266 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.062244    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0155c08f-058c-470c-af80-7f79013365a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0155c08f-058c-470c-af80-7f79013365a3" (UID: "0155c08f-058c-470c-af80-7f79013365a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:40:42.062639 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.062610    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-dshm" (OuterVolumeSpecName: "dshm") pod "0155c08f-058c-470c-af80-7f79013365a3" (UID: "0155c08f-058c-470c-af80-7f79013365a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:40:42.062639 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.062619    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0155c08f-058c-470c-af80-7f79013365a3-kube-api-access-xvbgq" (OuterVolumeSpecName: "kube-api-access-xvbgq") pod "0155c08f-058c-470c-af80-7f79013365a3" (UID: "0155c08f-058c-470c-af80-7f79013365a3"). InnerVolumeSpecName "kube-api-access-xvbgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:40:42.080401 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.080353    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0155c08f-058c-470c-af80-7f79013365a3" (UID: "0155c08f-058c-470c-af80-7f79013365a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:40:42.088000 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.087973    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0155c08f-058c-470c-af80-7f79013365a3" (UID: "0155c08f-058c-470c-af80-7f79013365a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:40:42.161447 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.161418    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-kserve-provision-location\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.161447 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.161445    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbgq\" (UniqueName: \"kubernetes.io/projected/0155c08f-058c-470c-af80-7f79013365a3-kube-api-access-xvbgq\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.161774 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.161456    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0155c08f-058c-470c-af80-7f79013365a3-cabundle-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.161774 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.161465    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.161774 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.161473    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0155c08f-058c-470c-af80-7f79013365a3-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.161774 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.161480    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.161774 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.161487    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0155c08f-058c-470c-af80-7f79013365a3-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:40:42.422362 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.422322    2551 generic.go:356] "Generic (PLEG): container finished" podID="0155c08f-058c-470c-af80-7f79013365a3" containerID="03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608" exitCode=0
Aug 01 00:40:42.422362 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.422353    2551 generic.go:356] "Generic (PLEG): container finished" podID="0155c08f-058c-470c-af80-7f79013365a3" containerID="f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166" exitCode=0
Aug 01 00:40:42.422569 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.422400    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"
Aug 01 00:40:42.422569 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.422406    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" event={"ID":"0155c08f-058c-470c-af80-7f79013365a3","Type":"ContainerDied","Data":"03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608"}
Aug 01 00:40:42.422569 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.422487    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" event={"ID":"0155c08f-058c-470c-af80-7f79013365a3","Type":"ContainerDied","Data":"f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166"}
Aug 01 00:40:42.422569 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.422508    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn" event={"ID":"0155c08f-058c-470c-af80-7f79013365a3","Type":"ContainerDied","Data":"5f05b2336d78b58bf4805e3dfa90c05ed5963db4ffd3a29bd3a3d449263db067"}
Aug 01 00:40:42.422569 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.422528    2551 scope.go:117] "RemoveContainer" containerID="03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608"
Aug 01 00:40:42.431158 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.431138    2551 scope.go:117] "RemoveContainer" containerID="7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07"
Aug 01 00:40:42.438271 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.438254    2551 scope.go:117] "RemoveContainer" containerID="f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166"
Aug 01 00:40:42.445188 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.445164    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"]
Aug 01 00:40:42.445396 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.445379    2551 scope.go:117] "RemoveContainer" containerID="03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608"
Aug 01 00:40:42.445649 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:40:42.445629    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608\": container with ID starting with 03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608 not found: ID does not exist" containerID="03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608"
Aug 01 00:40:42.445715 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.445657    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608"} err="failed to get container status \"03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608\": rpc error: code = NotFound desc = could not find container \"03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608\": container with ID starting with 03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608 not found: ID does not exist"
Aug 01 00:40:42.445715 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.445679    2551 scope.go:117] "RemoveContainer" containerID="7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07"
Aug 01 00:40:42.445970 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:40:42.445946    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07\": container with ID starting with 7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07 not found: ID does not exist" containerID="7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07"
Aug 01 00:40:42.446026 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.445982    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07"} err="failed to get container status \"7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07\": rpc error: code = NotFound desc = could not find container \"7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07\": container with ID starting with 7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07 not found: ID does not exist"
Aug 01 00:40:42.446069 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.446005    2551 scope.go:117] "RemoveContainer" containerID="f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166"
Aug 01 00:40:42.446433 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:40:42.446304    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166\": container with ID starting with f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166 not found: ID does not exist" containerID="f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166"
Aug 01 00:40:42.446433 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.446334    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166"} err="failed to get container status \"f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166\": rpc error: code = NotFound desc = could not find container \"f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166\": container with ID starting with f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166 not found: ID does not exist"
Aug 01 00:40:42.446433 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.446355    2551 scope.go:117] "RemoveContainer" containerID="03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608"
Aug 01 00:40:42.446873 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.446832    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608"} err="failed to get container status \"03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608\": rpc error: code = NotFound desc = could not find container \"03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608\": container with ID starting with 03ce398aec0ef7baff0b2a7428f391beb827f75f0a3cf29d188f386735351608 not found: ID does not exist"
Aug 01 00:40:42.446873 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.446869    2551 scope.go:117] "RemoveContainer" containerID="7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07"
Aug 01 00:40:42.447240 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.447213    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07"} err="failed to get container status \"7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07\": rpc error: code = NotFound desc = could not find container \"7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07\": container with ID starting with 7cf8452c9c537c2e64844ddc498e42d8c8c9e722ac874e1b0f8f107440c07e07 not found: ID does not exist"
Aug 01 00:40:42.447240 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.447239    2551 scope.go:117] "RemoveContainer" containerID="f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166"
Aug 01 00:40:42.447510 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.447486    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166"} err="failed to get container status \"f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166\": rpc error: code = NotFound desc = could not find container \"f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166\": container with ID starting with f2557b2185285cc3b63aa8f1c65ba913e3da7b7377c0fd0bdd8942486e8be166 not found: ID does not exist"
Aug 01 00:40:42.448536 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:42.448496    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6954df89d7jzhpn"]
Aug 01 00:40:43.947179 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:43.947147    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0155c08f-058c-470c-af80-7f79013365a3" path="/var/lib/kubelet/pods/0155c08f-058c-470c-af80-7f79013365a3/volumes"
Aug 01 00:40:46.997718 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:46.997677    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:40:47.100675 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:47.100614    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:40:52.112109 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:52.112073    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:40:55.719051 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.719008    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"]
Aug 01 00:40:55.719526 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.719510    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="llm-d-routing-sidecar"
Aug 01 00:40:55.719598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.719532    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="llm-d-routing-sidecar"
Aug 01 00:40:55.719598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.719550    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main"
Aug 01 00:40:55.719598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.719559    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main"
Aug 01 00:40:55.719598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.719578    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="953ac4e8-ba98-47f4-ac83-281cf2c53e5b" containerName="main"
Aug 01 00:40:55.719598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.719587    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="953ac4e8-ba98-47f4-ac83-281cf2c53e5b" containerName="main"
Aug 01 00:40:55.719856 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.719600    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="storage-initializer"
Aug 01 00:40:55.719856 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.719609    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="storage-initializer"
Aug 01 00:40:55.719856 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.719692    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="953ac4e8-ba98-47f4-ac83-281cf2c53e5b" containerName="main"
Aug 01 00:40:55.719856 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.719705    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="main"
Aug 01 00:40:55.719856 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.719715    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0155c08f-058c-470c-af80-7f79013365a3" containerName="llm-d-routing-sidecar"
Aug 01 00:40:55.723214 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.723191    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.726245 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.726223    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:40:55.726373 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.726347    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Aug 01 00:40:55.726510 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.726468    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Aug 01 00:40:55.726672 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.726651    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-77tph\""
Aug 01 00:40:55.726909 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.726884    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Aug 01 00:40:55.727019 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.726975    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-qjl5z\""
Aug 01 00:40:55.727019 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.726980    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Aug 01 00:40:55.737558 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.737532    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"]
Aug 01 00:40:55.770502 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.770467    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:40:55.882801 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.882764    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81279964-12fb-49ce-9810-94f66ef02736-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.883003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.882805    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-dshm\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.883003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.882872    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81279964-12fb-49ce-9810-94f66ef02736-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.883003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.882905    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.883003 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.882959    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.883231 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.883041    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-home\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.883231 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.883107    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-model-cache\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.883231 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.883141    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm7p9\" (UniqueName: \"kubernetes.io/projected/81279964-12fb-49ce-9810-94f66ef02736-kube-api-access-qm7p9\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.984440 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.984345    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81279964-12fb-49ce-9810-94f66ef02736-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.984440 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.984385    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-dshm\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.984440 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.984412    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81279964-12fb-49ce-9810-94f66ef02736-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.984726 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.984538    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.984726 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.984578    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.984726 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.984615    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-home\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.984726 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.984664    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-model-cache\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.984726 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.984694    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm7p9\" (UniqueName: \"kubernetes.io/projected/81279964-12fb-49ce-9810-94f66ef02736-kube-api-access-qm7p9\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.985052 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.984973    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.985119 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.985052    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-model-cache\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.985178 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.985105    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-home\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.985178 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.985142    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.985450 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.985426    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81279964-12fb-49ce-9810-94f66ef02736-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.986981 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.986953    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-dshm\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.987130 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.987108    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81279964-12fb-49ce-9810-94f66ef02736-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:55.996377 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:55.996353    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm7p9\" (UniqueName: \"kubernetes.io/projected/81279964-12fb-49ce-9810-94f66ef02736-kube-api-access-qm7p9\") pod \"custom-route-timeout-pd-test-kserve-57f77cf8-ddncb\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:56.003035 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.003007    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"]
Aug 01 00:40:56.006063 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.006043    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"
Aug 01 00:40:56.008563 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.008543    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-f8cn5\""
Aug 01 00:40:56.017342 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.017318    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"]
Aug 01 00:40:56.033320 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.033298    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:56.034139 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.034117    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:40:56.186306 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.186280    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"]
Aug 01 00:40:56.186384 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.186299    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qf49\" (UniqueName: \"kubernetes.io/projected/f506508d-90b2-4184-8f58-6706a35e25b4-kube-api-access-2qf49\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp\" (UID: \"f506508d-90b2-4184-8f58-6706a35e25b4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"
Aug 01 00:40:56.186384 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.186360    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f506508d-90b2-4184-8f58-6706a35e25b4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp\" (UID: \"f506508d-90b2-4184-8f58-6706a35e25b4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"
Aug 01 00:40:56.188432 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:40:56.188407    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81279964_12fb_49ce_9810_94f66ef02736.slice/crio-7424811b2815158473b3e8936780cda065b44f59345f4be9b5415f6a77547301 WatchSource:0}: Error finding container 7424811b2815158473b3e8936780cda065b44f59345f4be9b5415f6a77547301: Status 404 returned error can't find the container with id 7424811b2815158473b3e8936780cda065b44f59345f4be9b5415f6a77547301
Aug 01 00:40:56.287143 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.287109    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f506508d-90b2-4184-8f58-6706a35e25b4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp\" (UID: \"f506508d-90b2-4184-8f58-6706a35e25b4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"
Aug 01 00:40:56.287304 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.287218    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qf49\" (UniqueName: \"kubernetes.io/projected/f506508d-90b2-4184-8f58-6706a35e25b4-kube-api-access-2qf49\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp\" (UID: \"f506508d-90b2-4184-8f58-6706a35e25b4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"
Aug 01 00:40:56.289678 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.289651    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f506508d-90b2-4184-8f58-6706a35e25b4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp\" (UID: \"f506508d-90b2-4184-8f58-6706a35e25b4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"
Aug 01 00:40:56.294285 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.294261    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qf49\" (UniqueName: \"kubernetes.io/projected/f506508d-90b2-4184-8f58-6706a35e25b4-kube-api-access-2qf49\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp\" (UID: \"f506508d-90b2-4184-8f58-6706a35e25b4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"
Aug 01 00:40:56.321217 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.321176    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"
Aug 01 00:40:56.447213 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.447188    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"]
Aug 01 00:40:56.449582 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:40:56.449553    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf506508d_90b2_4184_8f58_6706a35e25b4.slice/crio-441d6c07a19643d78b8b7ab350f8f623a8c44008b44c25e31da71bb48f19171c WatchSource:0}: Error finding container 441d6c07a19643d78b8b7ab350f8f623a8c44008b44c25e31da71bb48f19171c: Status 404 returned error can't find the container with id 441d6c07a19643d78b8b7ab350f8f623a8c44008b44c25e31da71bb48f19171c
Aug 01 00:40:56.483100 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.483070    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp" event={"ID":"f506508d-90b2-4184-8f58-6706a35e25b4","Type":"ContainerStarted","Data":"441d6c07a19643d78b8b7ab350f8f623a8c44008b44c25e31da71bb48f19171c"}
Aug 01 00:40:56.484475 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.484449    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" event={"ID":"81279964-12fb-49ce-9810-94f66ef02736","Type":"ContainerStarted","Data":"8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a"}
Aug 01 00:40:56.484558 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.484497    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" event={"ID":"81279964-12fb-49ce-9810-94f66ef02736","Type":"ContainerStarted","Data":"7424811b2815158473b3e8936780cda065b44f59345f4be9b5415f6a77547301"}
Aug 01 00:40:56.484558 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:56.484532    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:40:57.490736 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:57.490695    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp" event={"ID":"f506508d-90b2-4184-8f58-6706a35e25b4","Type":"ContainerStarted","Data":"9e932c9cd482b5085de8b62dfa5a2ced7928e6036566d329ae4ea0c8146fdd37"}
Aug 01 00:40:57.491188 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:57.490759    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"
Aug 01 00:40:57.492412 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:57.492384    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" event={"ID":"81279964-12fb-49ce-9810-94f66ef02736","Type":"ContainerStarted","Data":"40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012"}
Aug 01 00:40:57.510064 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:57.510003    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp" podStartSLOduration=2.508760647 podStartE2EDuration="2.508760647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:40:57.505736194 +0000 UTC m=+3300.199783610" watchObservedRunningTime="2026-08-01 00:40:57.508760647 +0000 UTC m=+3300.202808064"
Aug 01 00:40:58.079228 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:58.079188    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:40:58.179892 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:58.179860    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:40:59.506422 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:59.506370    2551 generic.go:356] "Generic (PLEG): container finished" podID="81279964-12fb-49ce-9810-94f66ef02736" containerID="40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012" exitCode=0
Aug 01 00:40:59.506889 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:40:59.506449    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" event={"ID":"81279964-12fb-49ce-9810-94f66ef02736","Type":"ContainerDied","Data":"40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012"}
Aug 01 00:41:00.513588 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:00.513546    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" event={"ID":"81279964-12fb-49ce-9810-94f66ef02736","Type":"ContainerStarted","Data":"796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc"}
Aug 01 00:41:00.538415 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:00.538354    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podStartSLOduration=3.301160678 podStartE2EDuration="5.538336697s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.237176019s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:41:00.537053774 +0000 UTC m=+3303.231101192" watchObservedRunningTime="2026-08-01 00:41:00.538336697 +0000 UTC m=+3303.232384113"
Aug 01 00:41:05.587632 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:05.587593    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:41:06.034352 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:06.034307    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:41:06.034562 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:06.034378    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:41:06.035770 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:06.035734    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8001/health\": dial tcp 10.133.0.75:8001: connect: connection refused"
Aug 01 00:41:06.047523 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:06.047496    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:41:15.033982 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:15.033944    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:41:15.135548 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:15.135513    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:41:16.039858 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:16.039804    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8001/health\": dial tcp 10.133.0.75:8001: connect: connection refused"
Aug 01 00:41:26.034744 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:26.034634    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8001/health\": dial tcp 10.133.0.75:8001: connect: connection refused"
Aug 01 00:41:28.498333 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:28.498303    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"
Aug 01 00:41:36.034093 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:36.034054    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8001/health\": dial tcp 10.133.0.75:8001: connect: connection refused"
Aug 01 00:41:37.962193 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:37.962159    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:41:38.060902 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:38.060858    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:41:46.033901 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:46.033848    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8001/health\": dial tcp 10.133.0.75:8001: connect: connection refused"
Aug 01 00:41:54.099693 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:54.099657    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:41:56.033664 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:56.033620    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8001/health\": dial tcp 10.133.0.75:8001: connect: connection refused"
Aug 01 00:41:57.749336 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:57.749293    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:41:57.848726 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:41:57.848660    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:42:01.692119 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:42:01.692077    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:42:01.955834 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:42:01.955742    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:42:04.229089 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:42:04.229041    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:42:06.034324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:42:06.034280    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8001/health\": dial tcp 10.133.0.75:8001: connect: connection refused"
Aug 01 00:42:09.679537 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:42:09.679499    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:42:09.780260 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:42:09.780214    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:42:16.034639 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:42:16.034591    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8001/health\": dial tcp 10.133.0.75:8001: connect: connection refused"
Aug 01 00:42:16.886163 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:42:16.886126    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:42:26.033759 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:42:26.033706    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8001/health\": dial tcp 10.133.0.75:8001: connect: connection refused"
Aug 01 00:42:36.034342 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:42:36.034285    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8001/health\": dial tcp 10.133.0.75:8001: connect: connection refused"
Aug 01 00:42:46.034350 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:42:46.034304    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8001/health\": dial tcp 10.133.0.75:8001: connect: connection refused"
Aug 01 00:42:56.044469 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:42:56.044381    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:42:56.056077 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:42:56.056053    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:43:04.905828 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:04.905795    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"]
Aug 01 00:43:04.908398 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:04.908373    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"]
Aug 01 00:43:04.909990 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:04.909954    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:43:06.361665 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:06.361615    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:43:08.498585 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:08.498542    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp" podUID="f506508d-90b2-4184-8f58-6706a35e25b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:43:18.499337 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:18.499296    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp" podUID="f506508d-90b2-4184-8f58-6706a35e25b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:43:19.925375 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:19.925337    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp" podUID="f506508d-90b2-4184-8f58-6706a35e25b4" containerName="main" containerID="cri-o://9e932c9cd482b5085de8b62dfa5a2ced7928e6036566d329ae4ea0c8146fdd37" gracePeriod=45
Aug 01 00:43:19.930613 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:19.930581    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main" containerID="cri-o://796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc" gracePeriod=45
Aug 01 00:43:20.173746 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.173721    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"
Aug 01 00:43:20.184546 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.184478    2551 generic.go:356] "Generic (PLEG): container finished" podID="f506508d-90b2-4184-8f58-6706a35e25b4" containerID="9e932c9cd482b5085de8b62dfa5a2ced7928e6036566d329ae4ea0c8146fdd37" exitCode=0
Aug 01 00:43:20.184546 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.184511    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"
Aug 01 00:43:20.184682 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.184563    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp" event={"ID":"f506508d-90b2-4184-8f58-6706a35e25b4","Type":"ContainerDied","Data":"9e932c9cd482b5085de8b62dfa5a2ced7928e6036566d329ae4ea0c8146fdd37"}
Aug 01 00:43:20.184682 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.184632    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp" event={"ID":"f506508d-90b2-4184-8f58-6706a35e25b4","Type":"ContainerDied","Data":"441d6c07a19643d78b8b7ab350f8f623a8c44008b44c25e31da71bb48f19171c"}
Aug 01 00:43:20.184682 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.184664    2551 scope.go:117] "RemoveContainer" containerID="9e932c9cd482b5085de8b62dfa5a2ced7928e6036566d329ae4ea0c8146fdd37"
Aug 01 00:43:20.198517 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.198489    2551 scope.go:117] "RemoveContainer" containerID="9e932c9cd482b5085de8b62dfa5a2ced7928e6036566d329ae4ea0c8146fdd37"
Aug 01 00:43:20.199141 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:43:20.199105    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e932c9cd482b5085de8b62dfa5a2ced7928e6036566d329ae4ea0c8146fdd37\": container with ID starting with 9e932c9cd482b5085de8b62dfa5a2ced7928e6036566d329ae4ea0c8146fdd37 not found: ID does not exist" containerID="9e932c9cd482b5085de8b62dfa5a2ced7928e6036566d329ae4ea0c8146fdd37"
Aug 01 00:43:20.199268 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.199150    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e932c9cd482b5085de8b62dfa5a2ced7928e6036566d329ae4ea0c8146fdd37"} err="failed to get container status \"9e932c9cd482b5085de8b62dfa5a2ced7928e6036566d329ae4ea0c8146fdd37\": rpc error: code = NotFound desc = could not find container \"9e932c9cd482b5085de8b62dfa5a2ced7928e6036566d329ae4ea0c8146fdd37\": container with ID starting with 9e932c9cd482b5085de8b62dfa5a2ced7928e6036566d329ae4ea0c8146fdd37 not found: ID does not exist"
Aug 01 00:43:20.296275 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.296238    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qf49\" (UniqueName: \"kubernetes.io/projected/f506508d-90b2-4184-8f58-6706a35e25b4-kube-api-access-2qf49\") pod \"f506508d-90b2-4184-8f58-6706a35e25b4\" (UID: \"f506508d-90b2-4184-8f58-6706a35e25b4\") "
Aug 01 00:43:20.296460 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.296293    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f506508d-90b2-4184-8f58-6706a35e25b4-tls-certs\") pod \"f506508d-90b2-4184-8f58-6706a35e25b4\" (UID: \"f506508d-90b2-4184-8f58-6706a35e25b4\") "
Aug 01 00:43:20.298552 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.298516    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f506508d-90b2-4184-8f58-6706a35e25b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f506508d-90b2-4184-8f58-6706a35e25b4" (UID: "f506508d-90b2-4184-8f58-6706a35e25b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:43:20.298662 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.298572    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f506508d-90b2-4184-8f58-6706a35e25b4-kube-api-access-2qf49" (OuterVolumeSpecName: "kube-api-access-2qf49") pod "f506508d-90b2-4184-8f58-6706a35e25b4" (UID: "f506508d-90b2-4184-8f58-6706a35e25b4"). InnerVolumeSpecName "kube-api-access-2qf49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:43:20.397728 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.397678    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f506508d-90b2-4184-8f58-6706a35e25b4-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:43:20.397728 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.397722    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qf49\" (UniqueName: \"kubernetes.io/projected/f506508d-90b2-4184-8f58-6706a35e25b4-kube-api-access-2qf49\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:43:20.506536 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.506502    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"]
Aug 01 00:43:20.509856 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.509824    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-74d748j2lp"]
Aug 01 00:43:20.548362 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.548328    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="llm-d-routing-sidecar" containerID="cri-o://8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a" gracePeriod=45
Aug 01 00:43:20.785654 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.785627    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:43:20.902317 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.902279    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-home\") pod \"81279964-12fb-49ce-9810-94f66ef02736\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") "
Aug 01 00:43:20.902511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.902353    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm7p9\" (UniqueName: \"kubernetes.io/projected/81279964-12fb-49ce-9810-94f66ef02736-kube-api-access-qm7p9\") pod \"81279964-12fb-49ce-9810-94f66ef02736\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") "
Aug 01 00:43:20.902511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.902372    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-kserve-provision-location\") pod \"81279964-12fb-49ce-9810-94f66ef02736\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") "
Aug 01 00:43:20.902511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.902392    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81279964-12fb-49ce-9810-94f66ef02736-tls-certs\") pod \"81279964-12fb-49ce-9810-94f66ef02736\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") "
Aug 01 00:43:20.902511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.902425    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-dshm\") pod \"81279964-12fb-49ce-9810-94f66ef02736\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") "
Aug 01 00:43:20.902511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.902448    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-model-cache\") pod \"81279964-12fb-49ce-9810-94f66ef02736\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") "
Aug 01 00:43:20.902511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.902479    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81279964-12fb-49ce-9810-94f66ef02736-cabundle-cert\") pod \"81279964-12fb-49ce-9810-94f66ef02736\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") "
Aug 01 00:43:20.902511 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.902511    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-tmp-dir\") pod \"81279964-12fb-49ce-9810-94f66ef02736\" (UID: \"81279964-12fb-49ce-9810-94f66ef02736\") "
Aug 01 00:43:20.902892 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.902722    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-model-cache" (OuterVolumeSpecName: "model-cache") pod "81279964-12fb-49ce-9810-94f66ef02736" (UID: "81279964-12fb-49ce-9810-94f66ef02736"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:43:20.902892 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.902820    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:43:20.903144 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.903118    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-home" (OuterVolumeSpecName: "home") pod "81279964-12fb-49ce-9810-94f66ef02736" (UID: "81279964-12fb-49ce-9810-94f66ef02736"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:43:20.903228 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.903162    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81279964-12fb-49ce-9810-94f66ef02736-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81279964-12fb-49ce-9810-94f66ef02736" (UID: "81279964-12fb-49ce-9810-94f66ef02736"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:43:20.904740 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.904714    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81279964-12fb-49ce-9810-94f66ef02736-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81279964-12fb-49ce-9810-94f66ef02736" (UID: "81279964-12fb-49ce-9810-94f66ef02736"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:43:20.905173 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.905148    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81279964-12fb-49ce-9810-94f66ef02736-kube-api-access-qm7p9" (OuterVolumeSpecName: "kube-api-access-qm7p9") pod "81279964-12fb-49ce-9810-94f66ef02736" (UID: "81279964-12fb-49ce-9810-94f66ef02736"). InnerVolumeSpecName "kube-api-access-qm7p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:43:20.905271 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.905145    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-dshm" (OuterVolumeSpecName: "dshm") pod "81279964-12fb-49ce-9810-94f66ef02736" (UID: "81279964-12fb-49ce-9810-94f66ef02736"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:43:20.921335 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.921299    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81279964-12fb-49ce-9810-94f66ef02736" (UID: "81279964-12fb-49ce-9810-94f66ef02736"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:43:20.931602 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:20.931576    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81279964-12fb-49ce-9810-94f66ef02736" (UID: "81279964-12fb-49ce-9810-94f66ef02736"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:43:21.004180 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.004156    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:43:21.004291 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.004181    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm7p9\" (UniqueName: \"kubernetes.io/projected/81279964-12fb-49ce-9810-94f66ef02736-kube-api-access-qm7p9\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:43:21.004291 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.004192    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-kserve-provision-location\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:43:21.004291 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.004201    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81279964-12fb-49ce-9810-94f66ef02736-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:43:21.004291 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.004209    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:43:21.004291 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.004218    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81279964-12fb-49ce-9810-94f66ef02736-cabundle-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:43:21.004291 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.004245    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81279964-12fb-49ce-9810-94f66ef02736-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:43:21.191679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.191641    2551 generic.go:356] "Generic (PLEG): container finished" podID="81279964-12fb-49ce-9810-94f66ef02736" containerID="796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc" exitCode=0
Aug 01 00:43:21.191679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.191669    2551 generic.go:356] "Generic (PLEG): container finished" podID="81279964-12fb-49ce-9810-94f66ef02736" containerID="8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a" exitCode=0
Aug 01 00:43:21.191889 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.191694    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" event={"ID":"81279964-12fb-49ce-9810-94f66ef02736","Type":"ContainerDied","Data":"796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc"}
Aug 01 00:43:21.191889 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.191747    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" event={"ID":"81279964-12fb-49ce-9810-94f66ef02736","Type":"ContainerDied","Data":"8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a"}
Aug 01 00:43:21.191889 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.191766    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb" event={"ID":"81279964-12fb-49ce-9810-94f66ef02736","Type":"ContainerDied","Data":"7424811b2815158473b3e8936780cda065b44f59345f4be9b5415f6a77547301"}
Aug 01 00:43:21.191889 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.191702    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"
Aug 01 00:43:21.191889 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.191788    2551 scope.go:117] "RemoveContainer" containerID="796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc"
Aug 01 00:43:21.200828 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.200811    2551 scope.go:117] "RemoveContainer" containerID="40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012"
Aug 01 00:43:21.208349 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.208329    2551 scope.go:117] "RemoveContainer" containerID="8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a"
Aug 01 00:43:21.215089 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.215065    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"]
Aug 01 00:43:21.216274 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.216254    2551 scope.go:117] "RemoveContainer" containerID="796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc"
Aug 01 00:43:21.216519 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:43:21.216500    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc\": container with ID starting with 796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc not found: ID does not exist" containerID="796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc"
Aug 01 00:43:21.216598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.216532    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc"} err="failed to get container status \"796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc\": rpc error: code = NotFound desc = could not find container \"796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc\": container with ID starting with 796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc not found: ID does not exist"
Aug 01 00:43:21.216598 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.216556    2551 scope.go:117] "RemoveContainer" containerID="40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012"
Aug 01 00:43:21.216823 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:43:21.216805    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012\": container with ID starting with 40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012 not found: ID does not exist" containerID="40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012"
Aug 01 00:43:21.216872 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.216829    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012"} err="failed to get container status \"40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012\": rpc error: code = NotFound desc = could not find container \"40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012\": container with ID starting with 40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012 not found: ID does not exist"
Aug 01 00:43:21.216872 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.216845    2551 scope.go:117] "RemoveContainer" containerID="8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a"
Aug 01 00:43:21.217140 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:43:21.217090    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a\": container with ID starting with 8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a not found: ID does not exist" containerID="8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a"
Aug 01 00:43:21.217140 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.217124    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a"} err="failed to get container status \"8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a\": rpc error: code = NotFound desc = could not find container \"8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a\": container with ID starting with 8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a not found: ID does not exist"
Aug 01 00:43:21.217140 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.217138    2551 scope.go:117] "RemoveContainer" containerID="796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc"
Aug 01 00:43:21.217400 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.217373    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc"} err="failed to get container status \"796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc\": rpc error: code = NotFound desc = could not find container \"796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc\": container with ID starting with 796dc2510ad17f95371f8fc6adb8a9a7f7d0a5f5c70329d56545cfcb5ccffcdc not found: ID does not exist"
Aug 01 00:43:21.217508 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.217402    2551 scope.go:117] "RemoveContainer" containerID="40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012"
Aug 01 00:43:21.218252 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.218203    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012"} err="failed to get container status \"40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012\": rpc error: code = NotFound desc = could not find container \"40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012\": container with ID starting with 40bcdc5e5d920024f742fbdbc534e8330335447d0ccd0249f1907a5804027012 not found: ID does not exist"
Aug 01 00:43:21.218252 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.218226    2551 scope.go:117] "RemoveContainer" containerID="8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a"
Aug 01 00:43:21.218468 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.218447    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a"} err="failed to get container status \"8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a\": rpc error: code = NotFound desc = could not find container \"8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a\": container with ID starting with 8f22e57848bf778e03b894d30969e0834b198c92a4c3c2b4112936c21ac5679a not found: ID does not exist"
Aug 01 00:43:21.219759 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.219742    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57f77cf8-ddncb"]
Aug 01 00:43:21.946938 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.946901    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81279964-12fb-49ce-9810-94f66ef02736" path="/var/lib/kubelet/pods/81279964-12fb-49ce-9810-94f66ef02736/volumes"
Aug 01 00:43:21.947372 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:21.947357    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f506508d-90b2-4184-8f58-6706a35e25b4" path="/var/lib/kubelet/pods/f506508d-90b2-4184-8f58-6706a35e25b4/volumes"
Aug 01 00:43:24.974973 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:24.974937    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:43:25.074338 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.074302    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:43:25.173953 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.173901    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"]
Aug 01 00:43:25.174701 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.174673    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f506508d-90b2-4184-8f58-6706a35e25b4" containerName="main"
Aug 01 00:43:25.174701 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.174702    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f506508d-90b2-4184-8f58-6706a35e25b4" containerName="main"
Aug 01 00:43:25.174851 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.174722    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="llm-d-routing-sidecar"
Aug 01 00:43:25.174851 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.174731    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="llm-d-routing-sidecar"
Aug 01 00:43:25.174851 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.174745    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main"
Aug 01 00:43:25.174851 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.174754    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main"
Aug 01 00:43:25.174851 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.174783    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="storage-initializer"
Aug 01 00:43:25.174851 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.174793    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="storage-initializer"
Aug 01 00:43:25.175063 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.174886    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f506508d-90b2-4184-8f58-6706a35e25b4" containerName="main"
Aug 01 00:43:25.175063 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.174900    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="llm-d-routing-sidecar"
Aug 01 00:43:25.175063 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.174911    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="81279964-12fb-49ce-9810-94f66ef02736" containerName="main"
Aug 01 00:43:25.180020 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.179986    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.182163 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.182136    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Aug 01 00:43:25.182163 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.182140    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-g87bw\""
Aug 01 00:43:25.182359 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.182222    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Aug 01 00:43:25.182359 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.182294    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Aug 01 00:43:25.190187 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.190163    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"]
Aug 01 00:43:25.207166 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.207138    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:43:25.237891 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.237814    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.237891 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.237846    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.237891 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.237867    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa33f700-c29b-460f-857a-10e6e83f1f61-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.238107 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.237940    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.238107 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.237977    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42c58\" (UniqueName: \"kubernetes.io/projected/fa33f700-c29b-460f-857a-10e6e83f1f61-kube-api-access-42c58\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.238107 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.238005    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.238107 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.238021    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.238107 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.238072    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.238293 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.238125    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.267671 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.267640    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:43:25.339054 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.339020    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.339247 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.339069    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42c58\" (UniqueName: \"kubernetes.io/projected/fa33f700-c29b-460f-857a-10e6e83f1f61-kube-api-access-42c58\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.339247 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.339104    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.339247 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.339122    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.339247 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.339143    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.339247 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.339183    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.339247 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.339204    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.339247 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.339232    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.339592 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.339259    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa33f700-c29b-460f-857a-10e6e83f1f61-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.339592 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.339519    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.339592 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.339560    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.339796 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.339777    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.339858 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.339845    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.340131 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.340112    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa33f700-c29b-460f-857a-10e6e83f1f61-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.341639 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.341614    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.341737 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.341708    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.346570 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.346550    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.346974 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.346952    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42c58\" (UniqueName: \"kubernetes.io/projected/fa33f700-c29b-460f-857a-10e6e83f1f61-kube-api-access-42c58\") pod \"router-gateway-2-openshift-default-6cb79f7856-vp7k7\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.492881 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.492767    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:25.626154 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.625971    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"]
Aug 01 00:43:25.628753 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:43:25.628723    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa33f700_c29b_460f_857a_10e6e83f1f61.slice/crio-b3daebb17ec6e85d97c9dd851b637a2ced963180fe72bf3853488cf4056b3391 WatchSource:0}: Error finding container b3daebb17ec6e85d97c9dd851b637a2ced963180fe72bf3853488cf4056b3391: Status 404 returned error can't find the container with id b3daebb17ec6e85d97c9dd851b637a2ced963180fe72bf3853488cf4056b3391
Aug 01 00:43:25.630814 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.630782    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:43:25.630899 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.630842    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:43:25.630899 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:25.630873    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:43:26.218448 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:26.218409    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7" event={"ID":"fa33f700-c29b-460f-857a-10e6e83f1f61","Type":"ContainerStarted","Data":"562675fa87e8c5f331483c1958f820580b05053fd2c0c1abbee5c5fd88df56c2"}
Aug 01 00:43:26.218836 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:26.218476    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7" event={"ID":"fa33f700-c29b-460f-857a-10e6e83f1f61","Type":"ContainerStarted","Data":"b3daebb17ec6e85d97c9dd851b637a2ced963180fe72bf3853488cf4056b3391"}
Aug 01 00:43:26.237298 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:26.237241    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7" podStartSLOduration=1.237221549 podStartE2EDuration="1.237221549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:43:26.236095373 +0000 UTC m=+3448.930142790" watchObservedRunningTime="2026-08-01 00:43:26.237221549 +0000 UTC m=+3448.931268965"
Aug 01 00:43:26.493659 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:26.493625    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:26.498471 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:26.498443    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:27.222957 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:27.222912    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:27.223800 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:27.223770    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:28.854482 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:28.854445    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:43:31.924752 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:31.924716    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:43:37.389555 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:37.389494    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:43:37.490104 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:37.490043    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:43:38.272105 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:38.272074    2551 generic.go:356] "Generic (PLEG): container finished" podID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerID="562675fa87e8c5f331483c1958f820580b05053fd2c0c1abbee5c5fd88df56c2" exitCode=0
Aug 01 00:43:38.272282 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:38.272114    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7" event={"ID":"fa33f700-c29b-460f-857a-10e6e83f1f61","Type":"ContainerDied","Data":"562675fa87e8c5f331483c1958f820580b05053fd2c0c1abbee5c5fd88df56c2"}
Aug 01 00:43:38.272495 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:38.272480    2551 scope.go:117] "RemoveContainer" containerID="562675fa87e8c5f331483c1958f820580b05053fd2c0c1abbee5c5fd88df56c2"
Aug 01 00:43:38.273352 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:38.273327    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:43:38.273405 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:38.273394    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:43:38.273437 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:38.273425    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Aug 01 00:43:39.278261 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:39.278225    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7" event={"ID":"fa33f700-c29b-460f-857a-10e6e83f1f61","Type":"ContainerStarted","Data":"5a4061eccfdf499ae6065817f4d4da2f29e0c29cd894c1eb0dd8030ccf6a848c"}
Aug 01 00:43:39.493082 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:39.493032    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:39.494282 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:39.494245    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7" podUID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Aug 01 00:43:40.493420 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:40.493375    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7" podUID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Aug 01 00:43:41.494152 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:41.494099    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7" podUID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Aug 01 00:43:42.222906 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:42.222834    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:42.494208 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:42.494107    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7" podUID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Aug 01 00:43:43.494352 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:43.494304    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7" podUID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Aug 01 00:43:44.519265 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:44.519225    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:44.520311 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:44.520285    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:43:58.969052 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:58.969006    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"]
Aug 01 00:43:58.972243 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:58.972221    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:58.974504 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:58.974479    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Aug 01 00:43:58.974710 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:58.974696    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-mf58h\""
Aug 01 00:43:58.974795 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:58.974774    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Aug 01 00:43:58.974852 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:58.974774    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-9hmc9\""
Aug 01 00:43:58.975163 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:58.975142    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:43:58.986806 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:58.986778    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"]
Aug 01 00:43:59.039791 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.039757    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18fa4ba2-ab59-4f0b-9395-96003e437ea3-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.039998 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.039807    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fa4ba2-ab59-4f0b-9395-96003e437ea3-tls-certs\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.039998 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.039837    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-tmp-dir\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.039998 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.039932    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-home\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.039998 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.039961    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-model-cache\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.039998 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.039996    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcrg4\" (UniqueName: \"kubernetes.io/projected/18fa4ba2-ab59-4f0b-9395-96003e437ea3-kube-api-access-hcrg4\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.040214 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.040042    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-dshm\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.040214 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.040075    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.040937 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.040888    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:43:59.141167 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.141122    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.141368 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.141291    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18fa4ba2-ab59-4f0b-9395-96003e437ea3-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.141368 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.141363    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fa4ba2-ab59-4f0b-9395-96003e437ea3-tls-certs\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.141496 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.141407    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-tmp-dir\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.141566 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.141556    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-home\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.141627 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.141581    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-model-cache\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.141681 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.141619    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.141681 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.141632    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcrg4\" (UniqueName: \"kubernetes.io/projected/18fa4ba2-ab59-4f0b-9395-96003e437ea3-kube-api-access-hcrg4\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.141782 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.141693    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-dshm\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.142119 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.141875    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-home\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.142119 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.141955    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-tmp-dir\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.142119 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.142030    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18fa4ba2-ab59-4f0b-9395-96003e437ea3-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.142426 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.142171    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-model-cache\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.144174 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.144150    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-dshm\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.144331 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.144312    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fa4ba2-ab59-4f0b-9395-96003e437ea3-tls-certs\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.149312 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.149284    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcrg4\" (UniqueName: \"kubernetes.io/projected/18fa4ba2-ab59-4f0b-9395-96003e437ea3-kube-api-access-hcrg4\") pod \"router-with-refs-pd-test-kserve-748cc697db-6qtlw\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.282675 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.282572    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:43:59.337999 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.337964    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"]
Aug 01 00:43:59.340624 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.340600    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"
Aug 01 00:43:59.343654 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.343321    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-wp5cp\""
Aug 01 00:43:59.350475 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.350399    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"]
Aug 01 00:43:59.427682 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.427657    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"]
Aug 01 00:43:59.429714 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:43:59.429686    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fa4ba2_ab59_4f0b_9395_96003e437ea3.slice/crio-bc28ca69a1c6a8b016c696334ca9946cf6b7c7514a255584278ce59ed0b1c518 WatchSource:0}: Error finding container bc28ca69a1c6a8b016c696334ca9946cf6b7c7514a255584278ce59ed0b1c518: Status 404 returned error can't find the container with id bc28ca69a1c6a8b016c696334ca9946cf6b7c7514a255584278ce59ed0b1c518
Aug 01 00:43:59.444370 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.444344    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70d734f-1a31-4450-a483-837abc872df8-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47\" (UID: \"f70d734f-1a31-4450-a483-837abc872df8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"
Aug 01 00:43:59.444474 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.444378    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngzst\" (UniqueName: \"kubernetes.io/projected/f70d734f-1a31-4450-a483-837abc872df8-kube-api-access-ngzst\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47\" (UID: \"f70d734f-1a31-4450-a483-837abc872df8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"
Aug 01 00:43:59.545279 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.545180    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70d734f-1a31-4450-a483-837abc872df8-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47\" (UID: \"f70d734f-1a31-4450-a483-837abc872df8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"
Aug 01 00:43:59.545279 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.545222    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngzst\" (UniqueName: \"kubernetes.io/projected/f70d734f-1a31-4450-a483-837abc872df8-kube-api-access-ngzst\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47\" (UID: \"f70d734f-1a31-4450-a483-837abc872df8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"
Aug 01 00:43:59.546613 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.546582    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:43:59.547895 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.547874    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70d734f-1a31-4450-a483-837abc872df8-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47\" (UID: \"f70d734f-1a31-4450-a483-837abc872df8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"
Aug 01 00:43:59.552974 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.552894    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngzst\" (UniqueName: \"kubernetes.io/projected/f70d734f-1a31-4450-a483-837abc872df8-kube-api-access-ngzst\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47\" (UID: \"f70d734f-1a31-4450-a483-837abc872df8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"
Aug 01 00:43:59.656764 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.656718    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"
Aug 01 00:43:59.658039 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.658004    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:43:59.797318 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:43:59.797285    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"]
Aug 01 00:43:59.799055 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:43:59.799018    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70d734f_1a31_4450_a483_837abc872df8.slice/crio-b1106b24568d33019161a1a2057f3b326c4dc08a5223eb2a6a12d9c81ceec7ab WatchSource:0}: Error finding container b1106b24568d33019161a1a2057f3b326c4dc08a5223eb2a6a12d9c81ceec7ab: Status 404 returned error can't find the container with id b1106b24568d33019161a1a2057f3b326c4dc08a5223eb2a6a12d9c81ceec7ab
Aug 01 00:44:00.374588 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:00.374545    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" event={"ID":"18fa4ba2-ab59-4f0b-9395-96003e437ea3","Type":"ContainerStarted","Data":"eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b"}
Aug 01 00:44:00.375072 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:00.374613    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" event={"ID":"18fa4ba2-ab59-4f0b-9395-96003e437ea3","Type":"ContainerStarted","Data":"bc28ca69a1c6a8b016c696334ca9946cf6b7c7514a255584278ce59ed0b1c518"}
Aug 01 00:44:00.375072 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:00.374642    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:44:00.376173 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:00.376145    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47" event={"ID":"f70d734f-1a31-4450-a483-837abc872df8","Type":"ContainerStarted","Data":"06b2f47492de8d52e0c1e620c348ba6ece977e270a399888999fb7e46c282458"}
Aug 01 00:44:00.376270 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:00.376190    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47" event={"ID":"f70d734f-1a31-4450-a483-837abc872df8","Type":"ContainerStarted","Data":"b1106b24568d33019161a1a2057f3b326c4dc08a5223eb2a6a12d9c81ceec7ab"}
Aug 01 00:44:00.376270 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:00.376230    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"
Aug 01 00:44:00.410324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:00.410264    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47" podStartSLOduration=1.410247259 podStartE2EDuration="1.410247259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:44:00.408618173 +0000 UTC m=+3483.102665590" watchObservedRunningTime="2026-08-01 00:44:00.410247259 +0000 UTC m=+3483.104294674"
Aug 01 00:44:01.382973 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:01.382914    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" event={"ID":"18fa4ba2-ab59-4f0b-9395-96003e437ea3","Type":"ContainerStarted","Data":"3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4"}
Aug 01 00:44:03.392890 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:03.392849    2551 generic.go:356] "Generic (PLEG): container finished" podID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerID="3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4" exitCode=0
Aug 01 00:44:03.393317 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:03.392932    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" event={"ID":"18fa4ba2-ab59-4f0b-9395-96003e437ea3","Type":"ContainerDied","Data":"3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4"}
Aug 01 00:44:04.399855 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:04.399812    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" event={"ID":"18fa4ba2-ab59-4f0b-9395-96003e437ea3","Type":"ContainerStarted","Data":"d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98"}
Aug 01 00:44:04.421637 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:04.421574    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podStartSLOduration=4.326478079 podStartE2EDuration="6.421557464s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.095079385s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:44:04.420170572 +0000 UTC m=+3487.114217988" watchObservedRunningTime="2026-08-01 00:44:04.421557464 +0000 UTC m=+3487.115604878"
Aug 01 00:44:09.283344 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:09.283304    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:44:09.284019 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:09.283406    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:44:09.284716 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:09.284678    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8001/health\": dial tcp 10.133.0.78:8001: connect: connection refused"
Aug 01 00:44:09.296932 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:09.296898    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:44:19.283251 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:19.283193    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8001/health\": dial tcp 10.133.0.78:8001: connect: connection refused"
Aug 01 00:44:29.283309 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:29.283180    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8001/health\": dial tcp 10.133.0.78:8001: connect: connection refused"
Aug 01 00:44:31.385204 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:31.385171    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"
Aug 01 00:44:32.225491 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:32.225453    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:44:32.285089 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:32.285057    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:44:39.283769 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:39.283721    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8001/health\": dial tcp 10.133.0.78:8001: connect: connection refused"
Aug 01 00:44:41.385729 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:41.385695    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:44:49.283191 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:49.283138    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8001/health\": dial tcp 10.133.0.78:8001: connect: connection refused"
Aug 01 00:44:59.283807 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:44:59.283753    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8001/health\": dial tcp 10.133.0.78:8001: connect: connection refused"
Aug 01 00:45:09.283815 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:09.283757    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8001/health\": dial tcp 10.133.0.78:8001: connect: connection refused"
Aug 01 00:45:19.283345 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:19.283293    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8001/health\": dial tcp 10.133.0.78:8001: connect: connection refused"
Aug 01 00:45:29.284104 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:29.284031    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8001/health\": dial tcp 10.133.0.78:8001: connect: connection refused"
Aug 01 00:45:37.133169 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:37.133127    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:45:37.222740 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:37.222704    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:45:39.283352 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:39.283303    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8001/health\": dial tcp 10.133.0.78:8001: connect: connection refused"
Aug 01 00:45:42.939931 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:42.939877    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:45:44.577461 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:44.577410    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:45:44.836424 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:44.836337    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:45:48.528418 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:48.528384    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:45:48.596321 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:48.596281    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:45:49.283326 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:49.283267    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8001/health\": dial tcp 10.133.0.78:8001: connect: connection refused"
Aug 01 00:45:56.036823 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:56.036787    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:45:59.293373 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:59.293339    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:45:59.312847 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:45:59.312814    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:46:08.069785 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:08.069751    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"]
Aug 01 00:46:08.070937 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:08.070887    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:46:08.075673 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:08.075648    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"]
Aug 01 00:46:08.133127 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:08.133097    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:46:11.384574 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:11.384533    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47" podUID="f70d734f-1a31-4450-a483-837abc872df8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:46:21.385254 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:21.385214    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47" podUID="f70d734f-1a31-4450-a483-837abc872df8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 01 00:46:22.263515 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:22.263462    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:46:23.089662 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:23.089611    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47" podUID="f70d734f-1a31-4450-a483-837abc872df8" containerName="main" containerID="cri-o://06b2f47492de8d52e0c1e620c348ba6ece977e270a399888999fb7e46c282458" gracePeriod=45
Aug 01 00:46:23.098259 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:23.098221    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main" containerID="cri-o://d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98" gracePeriod=45
Aug 01 00:46:23.337782 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:23.337756    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"
Aug 01 00:46:23.460875 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:23.460782    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70d734f-1a31-4450-a483-837abc872df8-tls-certs\") pod \"f70d734f-1a31-4450-a483-837abc872df8\" (UID: \"f70d734f-1a31-4450-a483-837abc872df8\") "
Aug 01 00:46:23.460875 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:23.460845    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngzst\" (UniqueName: \"kubernetes.io/projected/f70d734f-1a31-4450-a483-837abc872df8-kube-api-access-ngzst\") pod \"f70d734f-1a31-4450-a483-837abc872df8\" (UID: \"f70d734f-1a31-4450-a483-837abc872df8\") "
Aug 01 00:46:23.463123 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:23.463092    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70d734f-1a31-4450-a483-837abc872df8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f70d734f-1a31-4450-a483-837abc872df8" (UID: "f70d734f-1a31-4450-a483-837abc872df8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:46:23.463286 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:23.463261    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70d734f-1a31-4450-a483-837abc872df8-kube-api-access-ngzst" (OuterVolumeSpecName: "kube-api-access-ngzst") pod "f70d734f-1a31-4450-a483-837abc872df8" (UID: "f70d734f-1a31-4450-a483-837abc872df8"). InnerVolumeSpecName "kube-api-access-ngzst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:46:23.562634 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:23.562598    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70d734f-1a31-4450-a483-837abc872df8-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:23.562634 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:23.562629    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngzst\" (UniqueName: \"kubernetes.io/projected/f70d734f-1a31-4450-a483-837abc872df8-kube-api-access-ngzst\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:23.718283 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:23.718242    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="llm-d-routing-sidecar" containerID="cri-o://eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b" gracePeriod=45
Aug 01 00:46:23.952659 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:23.952614    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:46:24.059625 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.059586    2551 generic.go:356] "Generic (PLEG): container finished" podID="f70d734f-1a31-4450-a483-837abc872df8" containerID="06b2f47492de8d52e0c1e620c348ba6ece977e270a399888999fb7e46c282458" exitCode=0
Aug 01 00:46:24.059815 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.059634    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"
Aug 01 00:46:24.059815 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.059673    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47" event={"ID":"f70d734f-1a31-4450-a483-837abc872df8","Type":"ContainerDied","Data":"06b2f47492de8d52e0c1e620c348ba6ece977e270a399888999fb7e46c282458"}
Aug 01 00:46:24.059815 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.059731    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47" event={"ID":"f70d734f-1a31-4450-a483-837abc872df8","Type":"ContainerDied","Data":"b1106b24568d33019161a1a2057f3b326c4dc08a5223eb2a6a12d9c81ceec7ab"}
Aug 01 00:46:24.059815 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.059761    2551 scope.go:117] "RemoveContainer" containerID="06b2f47492de8d52e0c1e620c348ba6ece977e270a399888999fb7e46c282458"
Aug 01 00:46:24.062335 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.062307    2551 generic.go:356] "Generic (PLEG): container finished" podID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerID="d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98" exitCode=0
Aug 01 00:46:24.062335 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.062333    2551 generic.go:356] "Generic (PLEG): container finished" podID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerID="eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b" exitCode=0
Aug 01 00:46:24.062524 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.062367    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" event={"ID":"18fa4ba2-ab59-4f0b-9395-96003e437ea3","Type":"ContainerDied","Data":"d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98"}
Aug 01 00:46:24.062524 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.062402    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"
Aug 01 00:46:24.062524 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.062419    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" event={"ID":"18fa4ba2-ab59-4f0b-9395-96003e437ea3","Type":"ContainerDied","Data":"eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b"}
Aug 01 00:46:24.062524 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.062436    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw" event={"ID":"18fa4ba2-ab59-4f0b-9395-96003e437ea3","Type":"ContainerDied","Data":"bc28ca69a1c6a8b016c696334ca9946cf6b7c7514a255584278ce59ed0b1c518"}
Aug 01 00:46:24.067410 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.067386    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-model-cache\") pod \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") "
Aug 01 00:46:24.067516 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.067447    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-home\") pod \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") "
Aug 01 00:46:24.067516 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.067474    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcrg4\" (UniqueName: \"kubernetes.io/projected/18fa4ba2-ab59-4f0b-9395-96003e437ea3-kube-api-access-hcrg4\") pod \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") "
Aug 01 00:46:24.067516 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.067511    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-dshm\") pod \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") "
Aug 01 00:46:24.067645 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.067586    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-tmp-dir\") pod \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") "
Aug 01 00:46:24.067645 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.067634    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-kserve-provision-location\") pod \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") "
Aug 01 00:46:24.067753 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.067679    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18fa4ba2-ab59-4f0b-9395-96003e437ea3-cabundle-cert\") pod \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") "
Aug 01 00:46:24.067753 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.067702    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fa4ba2-ab59-4f0b-9395-96003e437ea3-tls-certs\") pod \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\" (UID: \"18fa4ba2-ab59-4f0b-9395-96003e437ea3\") "
Aug 01 00:46:24.067860 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.067762    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-model-cache" (OuterVolumeSpecName: "model-cache") pod "18fa4ba2-ab59-4f0b-9395-96003e437ea3" (UID: "18fa4ba2-ab59-4f0b-9395-96003e437ea3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:24.068688 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.068137    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.068688 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.068176    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-home" (OuterVolumeSpecName: "home") pod "18fa4ba2-ab59-4f0b-9395-96003e437ea3" (UID: "18fa4ba2-ab59-4f0b-9395-96003e437ea3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:24.068961 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.068912    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18fa4ba2-ab59-4f0b-9395-96003e437ea3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "18fa4ba2-ab59-4f0b-9395-96003e437ea3" (UID: "18fa4ba2-ab59-4f0b-9395-96003e437ea3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:46:24.069171 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.069126    2551 scope.go:117] "RemoveContainer" containerID="06b2f47492de8d52e0c1e620c348ba6ece977e270a399888999fb7e46c282458"
Aug 01 00:46:24.069501 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:46:24.069418    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b2f47492de8d52e0c1e620c348ba6ece977e270a399888999fb7e46c282458\": container with ID starting with 06b2f47492de8d52e0c1e620c348ba6ece977e270a399888999fb7e46c282458 not found: ID does not exist" containerID="06b2f47492de8d52e0c1e620c348ba6ece977e270a399888999fb7e46c282458"
Aug 01 00:46:24.069501 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.069451    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b2f47492de8d52e0c1e620c348ba6ece977e270a399888999fb7e46c282458"} err="failed to get container status \"06b2f47492de8d52e0c1e620c348ba6ece977e270a399888999fb7e46c282458\": rpc error: code = NotFound desc = could not find container \"06b2f47492de8d52e0c1e620c348ba6ece977e270a399888999fb7e46c282458\": container with ID starting with 06b2f47492de8d52e0c1e620c348ba6ece977e270a399888999fb7e46c282458 not found: ID does not exist"
Aug 01 00:46:24.069501 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.069471    2551 scope.go:117] "RemoveContainer" containerID="d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98"
Aug 01 00:46:24.070238 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.070213    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-dshm" (OuterVolumeSpecName: "dshm") pod "18fa4ba2-ab59-4f0b-9395-96003e437ea3" (UID: "18fa4ba2-ab59-4f0b-9395-96003e437ea3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:24.070238 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.070226    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fa4ba2-ab59-4f0b-9395-96003e437ea3-kube-api-access-hcrg4" (OuterVolumeSpecName: "kube-api-access-hcrg4") pod "18fa4ba2-ab59-4f0b-9395-96003e437ea3" (UID: "18fa4ba2-ab59-4f0b-9395-96003e437ea3"). InnerVolumeSpecName "kube-api-access-hcrg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:46:24.070811 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.070753    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18fa4ba2-ab59-4f0b-9395-96003e437ea3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18fa4ba2-ab59-4f0b-9395-96003e437ea3" (UID: "18fa4ba2-ab59-4f0b-9395-96003e437ea3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:46:24.074731 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.074701    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"]
Aug 01 00:46:24.077489 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.077464    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ff6zs47"]
Aug 01 00:46:24.078907 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.078885    2551 scope.go:117] "RemoveContainer" containerID="3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4"
Aug 01 00:46:24.083039 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.083016    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18fa4ba2-ab59-4f0b-9395-96003e437ea3" (UID: "18fa4ba2-ab59-4f0b-9395-96003e437ea3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:24.086680 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.086660    2551 scope.go:117] "RemoveContainer" containerID="eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b"
Aug 01 00:46:24.094763 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.094608    2551 scope.go:117] "RemoveContainer" containerID="d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98"
Aug 01 00:46:24.095011 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:46:24.094893    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98\": container with ID starting with d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98 not found: ID does not exist" containerID="d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98"
Aug 01 00:46:24.095011 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.094939    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98"} err="failed to get container status \"d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98\": rpc error: code = NotFound desc = could not find container \"d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98\": container with ID starting with d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98 not found: ID does not exist"
Aug 01 00:46:24.095011 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.094957    2551 scope.go:117] "RemoveContainer" containerID="3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4"
Aug 01 00:46:24.095205 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:46:24.095177    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4\": container with ID starting with 3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4 not found: ID does not exist" containerID="3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4"
Aug 01 00:46:24.095255 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.095211    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4"} err="failed to get container status \"3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4\": rpc error: code = NotFound desc = could not find container \"3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4\": container with ID starting with 3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4 not found: ID does not exist"
Aug 01 00:46:24.095255 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.095231    2551 scope.go:117] "RemoveContainer" containerID="eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b"
Aug 01 00:46:24.095347 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.095325    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18fa4ba2-ab59-4f0b-9395-96003e437ea3" (UID: "18fa4ba2-ab59-4f0b-9395-96003e437ea3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:24.095459 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:46:24.095442    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b\": container with ID starting with eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b not found: ID does not exist" containerID="eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b"
Aug 01 00:46:24.095506 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.095465    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b"} err="failed to get container status \"eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b\": rpc error: code = NotFound desc = could not find container \"eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b\": container with ID starting with eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b not found: ID does not exist"
Aug 01 00:46:24.095506 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.095481    2551 scope.go:117] "RemoveContainer" containerID="d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98"
Aug 01 00:46:24.095688 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.095671    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98"} err="failed to get container status \"d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98\": rpc error: code = NotFound desc = could not find container \"d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98\": container with ID starting with d38974453c03e9d76bcea059edbee6e0e31af75b3b799eff41d4a197e2981c98 not found: ID does not exist"
Aug 01 00:46:24.095733 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.095688    2551 scope.go:117] "RemoveContainer" containerID="3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4"
Aug 01 00:46:24.095897 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.095881    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4"} err="failed to get container status \"3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4\": rpc error: code = NotFound desc = could not find container \"3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4\": container with ID starting with 3de32619b4c085f74e27c7b1482b40a8f9c31672dbf81568f3554223c3578bc4 not found: ID does not exist"
Aug 01 00:46:24.095956 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.095897    2551 scope.go:117] "RemoveContainer" containerID="eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b"
Aug 01 00:46:24.096140 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.096118    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b"} err="failed to get container status \"eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b\": rpc error: code = NotFound desc = could not find container \"eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b\": container with ID starting with eba2c349d558f9efff3747e7baf04bf2ad405a80ce86bb9b1f9e9c0d88a2e74b not found: ID does not exist"
Aug 01 00:46:24.169434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.169387    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.169434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.169428    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-kserve-provision-location\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.169434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.169442    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18fa4ba2-ab59-4f0b-9395-96003e437ea3-cabundle-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.169747 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.169458    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fa4ba2-ab59-4f0b-9395-96003e437ea3-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.169747 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.169466    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.169747 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.169475    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcrg4\" (UniqueName: \"kubernetes.io/projected/18fa4ba2-ab59-4f0b-9395-96003e437ea3-kube-api-access-hcrg4\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.169747 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.169483    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18fa4ba2-ab59-4f0b-9395-96003e437ea3-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:24.384948 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.384901    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"]
Aug 01 00:46:24.388968 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:24.388943    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-748cc697db-6qtlw"]
Aug 01 00:46:25.951733 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:25.951687    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" path="/var/lib/kubelet/pods/18fa4ba2-ab59-4f0b-9395-96003e437ea3/volumes"
Aug 01 00:46:25.952568 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:25.952543    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f70d734f-1a31-4450-a483-837abc872df8" path="/var/lib/kubelet/pods/f70d734f-1a31-4450-a483-837abc872df8/volumes"
Aug 01 00:46:27.917046 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:27.917012    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:46:28.015966 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:28.015933    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:46:32.859705 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:32.859669    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"]
Aug 01 00:46:32.860547 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:32.860504    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7" podUID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerName="istio-proxy" containerID="cri-o://5a4061eccfdf499ae6065817f4d4da2f29e0c29cd894c1eb0dd8030ccf6a848c" gracePeriod=30
Aug 01 00:46:32.908212 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:32.908176    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:46:38.105987 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.105958    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:46:38.131026 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.130909    2551 generic.go:356] "Generic (PLEG): container finished" podID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerID="5a4061eccfdf499ae6065817f4d4da2f29e0c29cd894c1eb0dd8030ccf6a848c" exitCode=0
Aug 01 00:46:38.131026 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.130961    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7" event={"ID":"fa33f700-c29b-460f-857a-10e6e83f1f61","Type":"ContainerDied","Data":"5a4061eccfdf499ae6065817f4d4da2f29e0c29cd894c1eb0dd8030ccf6a848c"}
Aug 01 00:46:38.131026 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.130991    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"
Aug 01 00:46:38.131309 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.131031    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7" event={"ID":"fa33f700-c29b-460f-857a-10e6e83f1f61","Type":"ContainerDied","Data":"b3daebb17ec6e85d97c9dd851b637a2ced963180fe72bf3853488cf4056b3391"}
Aug 01 00:46:38.131309 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.131062    2551 scope.go:117] "RemoveContainer" containerID="5a4061eccfdf499ae6065817f4d4da2f29e0c29cd894c1eb0dd8030ccf6a848c"
Aug 01 00:46:38.142332 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.142307    2551 scope.go:117] "RemoveContainer" containerID="562675fa87e8c5f331483c1958f820580b05053fd2c0c1abbee5c5fd88df56c2"
Aug 01 00:46:38.151032 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.151008    2551 scope.go:117] "RemoveContainer" containerID="5a4061eccfdf499ae6065817f4d4da2f29e0c29cd894c1eb0dd8030ccf6a848c"
Aug 01 00:46:38.151349 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:46:38.151324    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a4061eccfdf499ae6065817f4d4da2f29e0c29cd894c1eb0dd8030ccf6a848c\": container with ID starting with 5a4061eccfdf499ae6065817f4d4da2f29e0c29cd894c1eb0dd8030ccf6a848c not found: ID does not exist" containerID="5a4061eccfdf499ae6065817f4d4da2f29e0c29cd894c1eb0dd8030ccf6a848c"
Aug 01 00:46:38.151447 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.151364    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a4061eccfdf499ae6065817f4d4da2f29e0c29cd894c1eb0dd8030ccf6a848c"} err="failed to get container status \"5a4061eccfdf499ae6065817f4d4da2f29e0c29cd894c1eb0dd8030ccf6a848c\": rpc error: code = NotFound desc = could not find container \"5a4061eccfdf499ae6065817f4d4da2f29e0c29cd894c1eb0dd8030ccf6a848c\": container with ID starting with 5a4061eccfdf499ae6065817f4d4da2f29e0c29cd894c1eb0dd8030ccf6a848c not found: ID does not exist"
Aug 01 00:46:38.151447 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.151390    2551 scope.go:117] "RemoveContainer" containerID="562675fa87e8c5f331483c1958f820580b05053fd2c0c1abbee5c5fd88df56c2"
Aug 01 00:46:38.151684 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:46:38.151659    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562675fa87e8c5f331483c1958f820580b05053fd2c0c1abbee5c5fd88df56c2\": container with ID starting with 562675fa87e8c5f331483c1958f820580b05053fd2c0c1abbee5c5fd88df56c2 not found: ID does not exist" containerID="562675fa87e8c5f331483c1958f820580b05053fd2c0c1abbee5c5fd88df56c2"
Aug 01 00:46:38.151727 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.151694    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562675fa87e8c5f331483c1958f820580b05053fd2c0c1abbee5c5fd88df56c2"} err="failed to get container status \"562675fa87e8c5f331483c1958f820580b05053fd2c0c1abbee5c5fd88df56c2\": rpc error: code = NotFound desc = could not find container \"562675fa87e8c5f331483c1958f820580b05053fd2c0c1abbee5c5fd88df56c2\": container with ID starting with 562675fa87e8c5f331483c1958f820580b05053fd2c0c1abbee5c5fd88df56c2 not found: ID does not exist"
Aug 01 00:46:38.201455 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.201422    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-podinfo\") pod \"fa33f700-c29b-460f-857a-10e6e83f1f61\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") "
Aug 01 00:46:38.201630 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.201472    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-credential-socket\") pod \"fa33f700-c29b-460f-857a-10e6e83f1f61\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") "
Aug 01 00:46:38.201630 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.201506    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa33f700-c29b-460f-857a-10e6e83f1f61-istiod-ca-cert\") pod \"fa33f700-c29b-460f-857a-10e6e83f1f61\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") "
Aug 01 00:46:38.201630 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.201526    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-workload-certs\") pod \"fa33f700-c29b-460f-857a-10e6e83f1f61\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") "
Aug 01 00:46:38.201630 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.201557    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-envoy\") pod \"fa33f700-c29b-460f-857a-10e6e83f1f61\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") "
Aug 01 00:46:38.201630 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.201581    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42c58\" (UniqueName: \"kubernetes.io/projected/fa33f700-c29b-460f-857a-10e6e83f1f61-kube-api-access-42c58\") pod \"fa33f700-c29b-460f-857a-10e6e83f1f61\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") "
Aug 01 00:46:38.201890 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.201635    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-data\") pod \"fa33f700-c29b-460f-857a-10e6e83f1f61\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") "
Aug 01 00:46:38.201890 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.201675    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-token\") pod \"fa33f700-c29b-460f-857a-10e6e83f1f61\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") "
Aug 01 00:46:38.201890 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.201706    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-workload-socket\") pod \"fa33f700-c29b-460f-857a-10e6e83f1f61\" (UID: \"fa33f700-c29b-460f-857a-10e6e83f1f61\") "
Aug 01 00:46:38.202084 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.201876    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fa33f700-c29b-460f-857a-10e6e83f1f61" (UID: "fa33f700-c29b-460f-857a-10e6e83f1f61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:38.202084 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.201996    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-credential-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:38.202084 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.202002    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa33f700-c29b-460f-857a-10e6e83f1f61-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fa33f700-c29b-460f-857a-10e6e83f1f61" (UID: "fa33f700-c29b-460f-857a-10e6e83f1f61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:46:38.202246 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.202100    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fa33f700-c29b-460f-857a-10e6e83f1f61" (UID: "fa33f700-c29b-460f-857a-10e6e83f1f61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:38.202301 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.202240    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-data" (OuterVolumeSpecName: "istio-data") pod "fa33f700-c29b-460f-857a-10e6e83f1f61" (UID: "fa33f700-c29b-460f-857a-10e6e83f1f61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:38.202358 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.202306    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fa33f700-c29b-460f-857a-10e6e83f1f61" (UID: "fa33f700-c29b-460f-857a-10e6e83f1f61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:38.203796 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.203766    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fa33f700-c29b-460f-857a-10e6e83f1f61" (UID: "fa33f700-c29b-460f-857a-10e6e83f1f61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 01 00:46:38.203976 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.203955    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-token" (OuterVolumeSpecName: "istio-token") pod "fa33f700-c29b-460f-857a-10e6e83f1f61" (UID: "fa33f700-c29b-460f-857a-10e6e83f1f61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:46:38.204055 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.203988    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa33f700-c29b-460f-857a-10e6e83f1f61-kube-api-access-42c58" (OuterVolumeSpecName: "kube-api-access-42c58") pod "fa33f700-c29b-460f-857a-10e6e83f1f61" (UID: "fa33f700-c29b-460f-857a-10e6e83f1f61"). InnerVolumeSpecName "kube-api-access-42c58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:46:38.204119 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.204079    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fa33f700-c29b-460f-857a-10e6e83f1f61" (UID: "fa33f700-c29b-460f-857a-10e6e83f1f61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:46:38.303136 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.303097    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-envoy\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:38.303136 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.303134    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42c58\" (UniqueName: \"kubernetes.io/projected/fa33f700-c29b-460f-857a-10e6e83f1f61-kube-api-access-42c58\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:38.303348 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.303148    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-data\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:38.303348 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.303160    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-token\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:38.303348 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.303172    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-workload-socket\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:38.303348 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.303184    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa33f700-c29b-460f-857a-10e6e83f1f61-istio-podinfo\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:38.303348 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.303194    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa33f700-c29b-460f-857a-10e6e83f1f61-istiod-ca-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:38.303348 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.303205    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa33f700-c29b-460f-857a-10e6e83f1f61-workload-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:46:38.343750 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.343368    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"]
Aug 01 00:46:38.343959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.343807    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="storage-initializer"
Aug 01 00:46:38.343959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.343826    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="storage-initializer"
Aug 01 00:46:38.343959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.343838    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main"
Aug 01 00:46:38.343959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.343846    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main"
Aug 01 00:46:38.343959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.343859    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerName="istio-proxy"
Aug 01 00:46:38.343959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.343867    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerName="istio-proxy"
Aug 01 00:46:38.343959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.343875    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70d734f-1a31-4450-a483-837abc872df8" containerName="main"
Aug 01 00:46:38.343959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.343883    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70d734f-1a31-4450-a483-837abc872df8" containerName="main"
Aug 01 00:46:38.343959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.343900    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerName="istio-proxy"
Aug 01 00:46:38.343959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.343908    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerName="istio-proxy"
Aug 01 00:46:38.343959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.343943    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="llm-d-routing-sidecar"
Aug 01 00:46:38.343959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.343952    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="llm-d-routing-sidecar"
Aug 01 00:46:38.344559 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.344050    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerName="istio-proxy"
Aug 01 00:46:38.344559 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.344063    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="llm-d-routing-sidecar"
Aug 01 00:46:38.344559 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.344072    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="18fa4ba2-ab59-4f0b-9395-96003e437ea3" containerName="main"
Aug 01 00:46:38.344559 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.344084    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70d734f-1a31-4450-a483-837abc872df8" containerName="main"
Aug 01 00:46:38.344559 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.344250    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa33f700-c29b-460f-857a-10e6e83f1f61" containerName="istio-proxy"
Aug 01 00:46:38.348811 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.348784    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.351090 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.351067    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:46:38.351215 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.351068    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Aug 01 00:46:38.351361 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.351346    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Aug 01 00:46:38.351443 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.351412    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Aug 01 00:46:38.351679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.351662    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Aug 01 00:46:38.351821 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.351802    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-vw5vf\""
Aug 01 00:46:38.359014 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.358988    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"]
Aug 01 00:46:38.377693 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.377657    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:46:38.403639 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.403528    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/771c394d-8204-4245-991b-f91a5ec5f6ba-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.403639 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.403577    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2zx8\" (UniqueName: \"kubernetes.io/projected/771c394d-8204-4245-991b-f91a5ec5f6ba-kube-api-access-j2zx8\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.403639 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.403612    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.404037 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.403642    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.404037 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.403668    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.404037 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.403723    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.404037 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.403788    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771c394d-8204-4245-991b-f91a5ec5f6ba-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.404037 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.403854    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.451632 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.451602    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"]
Aug 01 00:46:38.454878 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.454848    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-vp7k7"]
Aug 01 00:46:38.504224 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.504187    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771c394d-8204-4245-991b-f91a5ec5f6ba-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.504386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.504250    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.504386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.504291    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/771c394d-8204-4245-991b-f91a5ec5f6ba-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.504386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.504313    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2zx8\" (UniqueName: \"kubernetes.io/projected/771c394d-8204-4245-991b-f91a5ec5f6ba-kube-api-access-j2zx8\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.504386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.504341    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.504386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.504370    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.504657 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.504398    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.504657 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.504425    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.504767 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.504748    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.504831 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.504808    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.504892 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.504847    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.504964 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.504945    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.505250 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.505229    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/771c394d-8204-4245-991b-f91a5ec5f6ba-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.506909 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.506891    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771c394d-8204-4245-991b-f91a5ec5f6ba-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.506996 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.506980    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.512386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.512359    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2zx8\" (UniqueName: \"kubernetes.io/projected/771c394d-8204-4245-991b-f91a5ec5f6ba-kube-api-access-j2zx8\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.661651 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.661556    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:38.802469 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:38.802436    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"]
Aug 01 00:46:38.806098 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:46:38.806059    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771c394d_8204_4245_991b_f91a5ec5f6ba.slice/crio-945248f329137cff128efd32719b5051bb9ad3a3f05e03cc06a51538962fe6bd WatchSource:0}: Error finding container 945248f329137cff128efd32719b5051bb9ad3a3f05e03cc06a51538962fe6bd: Status 404 returned error can't find the container with id 945248f329137cff128efd32719b5051bb9ad3a3f05e03cc06a51538962fe6bd
Aug 01 00:46:39.137409 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:39.137373    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" event={"ID":"771c394d-8204-4245-991b-f91a5ec5f6ba","Type":"ContainerStarted","Data":"cc648b4b24535ba4507234397b4a0c4ceee0288c5e34fe285030634dcf487269"}
Aug 01 00:46:39.137880 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:39.137434    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" event={"ID":"771c394d-8204-4245-991b-f91a5ec5f6ba","Type":"ContainerStarted","Data":"945248f329137cff128efd32719b5051bb9ad3a3f05e03cc06a51538962fe6bd"}
Aug 01 00:46:39.947019 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:39.946988    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa33f700-c29b-460f-857a-10e6e83f1f61" path="/var/lib/kubelet/pods/fa33f700-c29b-460f-857a-10e6e83f1f61/volumes"
Aug 01 00:46:41.149330 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:41.149293    2551 generic.go:356] "Generic (PLEG): container finished" podID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerID="cc648b4b24535ba4507234397b4a0c4ceee0288c5e34fe285030634dcf487269" exitCode=0
Aug 01 00:46:41.149819 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:41.149373    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" event={"ID":"771c394d-8204-4245-991b-f91a5ec5f6ba","Type":"ContainerDied","Data":"cc648b4b24535ba4507234397b4a0c4ceee0288c5e34fe285030634dcf487269"}
Aug 01 00:46:41.650493 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:41.650449    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:46:41.747541 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:41.747507    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:46:42.155510 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:42.155467    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" event={"ID":"771c394d-8204-4245-991b-f91a5ec5f6ba","Type":"ContainerStarted","Data":"49fb60bf7457bd25f4050d525a18b265ddfee393b00fd98c213b2a40dd65ebb6"}
Aug 01 00:46:42.176659 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:42.176570    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podStartSLOduration=1.960393085 podStartE2EDuration="4.176551919s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.216158834s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:46:42.17461496 +0000 UTC m=+3644.868662369" watchObservedRunningTime="2026-08-01 00:46:42.176551919 +0000 UTC m=+3644.870599336"
Aug 01 00:46:42.661733 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:42.661688    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:48.662732 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:48.662682    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:46:48.664392 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:48.664361    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Aug 01 00:46:54.303170 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:54.303121    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:46:55.069517 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:55.069481    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:46:58.662398 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:46:58.662291    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Aug 01 00:47:00.673151 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:00.673113    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:47:00.725078 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:00.725038    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:47:06.620819 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:06.620776    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:47:08.662212 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:08.662167    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Aug 01 00:47:12.130148 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:12.130088    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:47:12.187876 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:12.187835    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:47:15.829572 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:15.829523    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:47:18.662363 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:18.662321    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Aug 01 00:47:23.544231 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:23.544121    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:47:23.645065 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:23.645014    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:47:28.662960 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:28.662892    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Aug 01 00:47:38.662936 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:38.662869    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Aug 01 00:47:48.663061 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:48.663000    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Aug 01 00:47:58.662878 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:47:58.662831    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Aug 01 00:48:08.662527 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:08.662483    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Aug 01 00:48:18.662627 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:18.662571    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Aug 01 00:48:28.662730 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:28.662685    2551 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Aug 01 00:48:38.673521 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:38.673485    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:48:38.681468 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:38.681447    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:48:43.162875 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:43.162839    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"]
Aug 01 00:48:43.184105 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:43.184075    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:48:44.496665 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:44.496632    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:48:58.185654 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:58.185613    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main" containerID="cri-o://49fb60bf7457bd25f4050d525a18b265ddfee393b00fd98c213b2a40dd65ebb6" gracePeriod=45
Aug 01 00:48:58.674665 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:58.674620    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Aug 01 00:48:58.939597 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:58.939567    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:48:59.077678 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.077641    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-dshm\") pod \"771c394d-8204-4245-991b-f91a5ec5f6ba\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") "
Aug 01 00:48:59.077678 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.077685    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2zx8\" (UniqueName: \"kubernetes.io/projected/771c394d-8204-4245-991b-f91a5ec5f6ba-kube-api-access-j2zx8\") pod \"771c394d-8204-4245-991b-f91a5ec5f6ba\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") "
Aug 01 00:48:59.077893 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.077702    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-home\") pod \"771c394d-8204-4245-991b-f91a5ec5f6ba\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") "
Aug 01 00:48:59.077893 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.077843    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-tmp-dir\") pod \"771c394d-8204-4245-991b-f91a5ec5f6ba\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") "
Aug 01 00:48:59.078014 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.077909    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/771c394d-8204-4245-991b-f91a5ec5f6ba-cabundle-cert\") pod \"771c394d-8204-4245-991b-f91a5ec5f6ba\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") "
Aug 01 00:48:59.078014 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.077949    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771c394d-8204-4245-991b-f91a5ec5f6ba-tls-certs\") pod \"771c394d-8204-4245-991b-f91a5ec5f6ba\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") "
Aug 01 00:48:59.078119 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.078088    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-kserve-provision-location\") pod \"771c394d-8204-4245-991b-f91a5ec5f6ba\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") "
Aug 01 00:48:59.078196 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.078178    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-model-cache\") pod \"771c394d-8204-4245-991b-f91a5ec5f6ba\" (UID: \"771c394d-8204-4245-991b-f91a5ec5f6ba\") "
Aug 01 00:48:59.078436 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.078391    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771c394d-8204-4245-991b-f91a5ec5f6ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "771c394d-8204-4245-991b-f91a5ec5f6ba" (UID: "771c394d-8204-4245-991b-f91a5ec5f6ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:48:59.078627 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.078606    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-home" (OuterVolumeSpecName: "home") pod "771c394d-8204-4245-991b-f91a5ec5f6ba" (UID: "771c394d-8204-4245-991b-f91a5ec5f6ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:48:59.078756 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.078649    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "771c394d-8204-4245-991b-f91a5ec5f6ba" (UID: "771c394d-8204-4245-991b-f91a5ec5f6ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:48:59.080672 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.080644    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/771c394d-8204-4245-991b-f91a5ec5f6ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "771c394d-8204-4245-991b-f91a5ec5f6ba" (UID: "771c394d-8204-4245-991b-f91a5ec5f6ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:48:59.080672 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.080661    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-dshm" (OuterVolumeSpecName: "dshm") pod "771c394d-8204-4245-991b-f91a5ec5f6ba" (UID: "771c394d-8204-4245-991b-f91a5ec5f6ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:48:59.080830 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.080757    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771c394d-8204-4245-991b-f91a5ec5f6ba-kube-api-access-j2zx8" (OuterVolumeSpecName: "kube-api-access-j2zx8") pod "771c394d-8204-4245-991b-f91a5ec5f6ba" (UID: "771c394d-8204-4245-991b-f91a5ec5f6ba"). InnerVolumeSpecName "kube-api-access-j2zx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:48:59.090452 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.090416    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "771c394d-8204-4245-991b-f91a5ec5f6ba" (UID: "771c394d-8204-4245-991b-f91a5ec5f6ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:48:59.103277 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.103242    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "771c394d-8204-4245-991b-f91a5ec5f6ba" (UID: "771c394d-8204-4245-991b-f91a5ec5f6ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:48:59.179214 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.179133    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-tmp-dir\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:48:59.179214 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.179167    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/771c394d-8204-4245-991b-f91a5ec5f6ba-cabundle-cert\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:48:59.179214 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.179177    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/771c394d-8204-4245-991b-f91a5ec5f6ba-tls-certs\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:48:59.179214 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.179186    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-kserve-provision-location\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:48:59.179214 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.179195    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-model-cache\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:48:59.179214 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.179205    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-dshm\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:48:59.179214 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.179213    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2zx8\" (UniqueName: \"kubernetes.io/projected/771c394d-8204-4245-991b-f91a5ec5f6ba-kube-api-access-j2zx8\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:48:59.179214 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.179220    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/771c394d-8204-4245-991b-f91a5ec5f6ba-home\") on node \"ip-10-0-131-136.ec2.internal\" DevicePath \"\""
Aug 01 00:48:59.750940 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.750883    2551 generic.go:356] "Generic (PLEG): container finished" podID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerID="49fb60bf7457bd25f4050d525a18b265ddfee393b00fd98c213b2a40dd65ebb6" exitCode=0
Aug 01 00:48:59.751385 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.750960    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" event={"ID":"771c394d-8204-4245-991b-f91a5ec5f6ba","Type":"ContainerDied","Data":"49fb60bf7457bd25f4050d525a18b265ddfee393b00fd98c213b2a40dd65ebb6"}
Aug 01 00:48:59.751385 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.750971    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"
Aug 01 00:48:59.751385 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.751066    2551 scope.go:117] "RemoveContainer" containerID="49fb60bf7457bd25f4050d525a18b265ddfee393b00fd98c213b2a40dd65ebb6"
Aug 01 00:48:59.751385 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.751053    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f" event={"ID":"771c394d-8204-4245-991b-f91a5ec5f6ba","Type":"ContainerDied","Data":"945248f329137cff128efd32719b5051bb9ad3a3f05e03cc06a51538962fe6bd"}
Aug 01 00:48:59.760362 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.760336    2551 scope.go:117] "RemoveContainer" containerID="cc648b4b24535ba4507234397b4a0c4ceee0288c5e34fe285030634dcf487269"
Aug 01 00:48:59.768386 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.768357    2551 scope.go:117] "RemoveContainer" containerID="49fb60bf7457bd25f4050d525a18b265ddfee393b00fd98c213b2a40dd65ebb6"
Aug 01 00:48:59.768716 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:48:59.768694    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49fb60bf7457bd25f4050d525a18b265ddfee393b00fd98c213b2a40dd65ebb6\": container with ID starting with 49fb60bf7457bd25f4050d525a18b265ddfee393b00fd98c213b2a40dd65ebb6 not found: ID does not exist" containerID="49fb60bf7457bd25f4050d525a18b265ddfee393b00fd98c213b2a40dd65ebb6"
Aug 01 00:48:59.768770 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.768727    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49fb60bf7457bd25f4050d525a18b265ddfee393b00fd98c213b2a40dd65ebb6"} err="failed to get container status \"49fb60bf7457bd25f4050d525a18b265ddfee393b00fd98c213b2a40dd65ebb6\": rpc error: code = NotFound desc = could not find container \"49fb60bf7457bd25f4050d525a18b265ddfee393b00fd98c213b2a40dd65ebb6\": container with ID starting with 49fb60bf7457bd25f4050d525a18b265ddfee393b00fd98c213b2a40dd65ebb6 not found: ID does not exist"
Aug 01 00:48:59.768770 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.768747    2551 scope.go:117] "RemoveContainer" containerID="cc648b4b24535ba4507234397b4a0c4ceee0288c5e34fe285030634dcf487269"
Aug 01 00:48:59.769035 ip-10-0-131-136 kubenswrapper[2551]: E0801 00:48:59.769004    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc648b4b24535ba4507234397b4a0c4ceee0288c5e34fe285030634dcf487269\": container with ID starting with cc648b4b24535ba4507234397b4a0c4ceee0288c5e34fe285030634dcf487269 not found: ID does not exist" containerID="cc648b4b24535ba4507234397b4a0c4ceee0288c5e34fe285030634dcf487269"
Aug 01 00:48:59.769138 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.769039    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc648b4b24535ba4507234397b4a0c4ceee0288c5e34fe285030634dcf487269"} err="failed to get container status \"cc648b4b24535ba4507234397b4a0c4ceee0288c5e34fe285030634dcf487269\": rpc error: code = NotFound desc = could not find container \"cc648b4b24535ba4507234397b4a0c4ceee0288c5e34fe285030634dcf487269\": container with ID starting with cc648b4b24535ba4507234397b4a0c4ceee0288c5e34fe285030634dcf487269 not found: ID does not exist"
Aug 01 00:48:59.774960 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.774929    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"]
Aug 01 00:48:59.780679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.780650    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f46f479cc2v48f"]
Aug 01 00:48:59.947118 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:48:59.947085    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" path="/var/lib/kubelet/pods/771c394d-8204-4245-991b-f91a5ec5f6ba/volumes"
Aug 01 00:49:03.184599 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:03.184564    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:49:06.845057 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:06.845023    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:49:06.943689 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:06.943656    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:49:08.964171 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:08.964139    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:49:14.372444 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:14.372406    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:49:14.433981 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:14.433904    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:49:29.955784 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:29.955750    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/0.log"
Aug 01 00:49:29.986225 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:29.986197    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/1.log"
Aug 01 00:49:30.934587 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:30.934552    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/0.log"
Aug 01 00:49:30.962429 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:30.962399    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/1.log"
Aug 01 00:49:31.881278 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:31.881240    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/0.log"
Aug 01 00:49:31.910492 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:31.910459    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/1.log"
Aug 01 00:49:32.852526 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:32.852489    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/0.log"
Aug 01 00:49:32.887770 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:32.887742    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/1.log"
Aug 01 00:49:33.903699 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:33.903661    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:33.957996 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:33.957966    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:35.225662 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:35.225623    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:35.486176 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:35.486089    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:37.170334 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:37.170303    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:37.231797 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:37.231769    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:38.564346 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:38.564314    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:38.627988 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:38.627959    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:39.851839 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:39.851809    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:39.916693 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:39.916662    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:41.165053 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:41.165013    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:41.231149 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:41.231114    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:42.476030 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:42.475990    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:42.535967 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:42.535935    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:43.766640 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:43.766605    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:43.831865 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:43.831835    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:45.072030 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:45.071981    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:45.129482 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:45.129454    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:46.386647 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:46.386610    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:46.449959 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:46.449908    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:47.708265 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:47.708217    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:47.762451 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:47.762424    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:49.005859 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:49.005819    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:49.077335 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:49.077304    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:50.328764 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:50.328728    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:50.389404 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:50.389374    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:51.626276 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:51.626238    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:51.687483 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:51.687456    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:52.918156 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:52.918119    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:52.993179 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:52.993152    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:54.229328 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:54.229290    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:54.288748 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:54.288718    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:55.528290 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:55.528246    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:55.586163 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:55.586134    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:56.808073 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:56.808034    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:56.868298 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:56.868266    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:58.121893 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:58.121863    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:58.177801 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:58.177772    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:49:59.420632 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:59.420593    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:49:59.484344 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:49:59.484304    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:50:00.718676 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:00.718638    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:50:00.779555 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:00.779529    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:50:02.038182 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:02.038149    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:50:02.094030 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:02.093995    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:50:03.365613 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:03.365576    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:50:03.427497 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:03.427468    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:50:04.700913 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:04.700660    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:50:04.766434 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:04.766403    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:50:06.027969 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:06.027934    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:50:06.082990 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:06.082965    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:50:08.301615 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:08.301574    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:50:08.466559 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:08.466521    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:50:08.532323 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:08.532292    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:50:11.134261 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:11.134229    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hfz7q_20d38ffc-045d-41cf-bcde-a74d3e0b8361/global-pull-secret-syncer/0.log"
Aug 01 00:50:11.226533 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:11.226495    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log"
Aug 01 00:50:11.355313 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:11.355276    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-136.ec2.internal_d4efdb1fd75703bb7b34b57a3634782a/haproxy/0.log"
Aug 01 00:50:13.301153 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:13.301115    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:50:13.369059 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:13.369021    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:50:13.933120 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:13.933081    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:50:14.075354 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:14.075318    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5z6n7_be479e0b-db97-4752-9f8a-4676b12319c5/konnectivity-agent/0.log" limit=1024
Aug 01 00:50:16.875932 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:16.875874    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-wmrp8_fa7b0e9f-0ba9-45cd-a560-07e454750004/authorino/0.log"
Aug 01 00:50:16.967209 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:16.967178    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ckq7h_1e174ef7-e0d1-4ea4-be22-062227293fde/kuadrant-console-plugin/0.log"
Aug 01 00:50:18.510408 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:18.510373    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6dbb86b874-t4lnc_904f772a-9e6f-4284-bb48-969e1f27c06c/metrics-server/0.log"
Aug 01 00:50:18.750679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:18.750650    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prc8z_a338d921-48df-4ec5-8f68-f4edccb201fd/node-exporter/0.log"
Aug 01 00:50:18.769679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:18.769600    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prc8z_a338d921-48df-4ec5-8f68-f4edccb201fd/kube-rbac-proxy/0.log"
Aug 01 00:50:18.802805 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:18.802768    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prc8z_a338d921-48df-4ec5-8f68-f4edccb201fd/init-textfile/0.log"
Aug 01 00:50:18.922744 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:18.922716    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c995521b-63c5-429f-9fee-2760aa423079/prometheus/0.log"
Aug 01 00:50:18.943877 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:18.943849    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c995521b-63c5-429f-9fee-2760aa423079/config-reloader/0.log"
Aug 01 00:50:18.964961 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:18.964933    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c995521b-63c5-429f-9fee-2760aa423079/thanos-sidecar/0.log"
Aug 01 00:50:18.985359 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:18.985333    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c995521b-63c5-429f-9fee-2760aa423079/kube-rbac-proxy-web/0.log"
Aug 01 00:50:19.005152 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:19.005128    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c995521b-63c5-429f-9fee-2760aa423079/kube-rbac-proxy/0.log"
Aug 01 00:50:19.024429 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:19.024357    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c995521b-63c5-429f-9fee-2760aa423079/kube-rbac-proxy-thanos/0.log"
Aug 01 00:50:19.048839 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:19.048811    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c995521b-63c5-429f-9fee-2760aa423079/init-config-reloader/0.log"
Aug 01 00:50:19.165545 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:19.165514    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-9nkh4_12b03887-d790-428b-9208-3100816be5d8/prometheus-operator-admission-webhook/0.log"
Aug 01 00:50:19.264358 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:19.264332    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-688b9bd9fb-nxrql_4a4f7f01-afc4-4b37-9046-856b4573b38e/thanos-query/0.log"
Aug 01 00:50:19.286901 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:19.286820    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-688b9bd9fb-nxrql_4a4f7f01-afc4-4b37-9046-856b4573b38e/kube-rbac-proxy-web/0.log"
Aug 01 00:50:19.308455 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:19.308425    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-688b9bd9fb-nxrql_4a4f7f01-afc4-4b37-9046-856b4573b38e/kube-rbac-proxy/0.log"
Aug 01 00:50:19.327560 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:19.327534    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-688b9bd9fb-nxrql_4a4f7f01-afc4-4b37-9046-856b4573b38e/prom-label-proxy/0.log"
Aug 01 00:50:19.347679 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:19.347654    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-688b9bd9fb-nxrql_4a4f7f01-afc4-4b37-9046-856b4573b38e/kube-rbac-proxy-rules/0.log"
Aug 01 00:50:19.368205 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:19.368173    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-688b9bd9fb-nxrql_4a4f7f01-afc4-4b37-9046-856b4573b38e/kube-rbac-proxy-metrics/0.log"
Aug 01 00:50:21.284095 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.284006    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"]
Aug 01 00:50:21.284477 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.284370    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main"
Aug 01 00:50:21.284477 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.284383    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main"
Aug 01 00:50:21.284477 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.284416    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="storage-initializer"
Aug 01 00:50:21.284477 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.284423    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="storage-initializer"
Aug 01 00:50:21.284477 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.284467    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="771c394d-8204-4245-991b-f91a5ec5f6ba" containerName="main"
Aug 01 00:50:21.287542 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.287518    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.289534 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.289508    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kbl7c\"/\"kube-root-ca.crt\""
Aug 01 00:50:21.289660 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.289562    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kbl7c\"/\"default-dockercfg-txszj\""
Aug 01 00:50:21.289660 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.289604    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kbl7c\"/\"openshift-service-ca.crt\""
Aug 01 00:50:21.295376 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.295352    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"]
Aug 01 00:50:21.327665 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.327633    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-sys\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.327807 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.327670    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-lib-modules\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.327807 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.327705    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-podres\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.327901 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.327803    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnm77\" (UniqueName: \"kubernetes.io/projected/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-kube-api-access-lnm77\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.327901 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.327857    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-proc\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.429230 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.429199    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-proc\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.429420 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.429244    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-sys\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.429420 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.429263    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-lib-modules\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.429420 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.429294    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-podres\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.429420 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.429341    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnm77\" (UniqueName: \"kubernetes.io/projected/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-kube-api-access-lnm77\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.429420 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.429348    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-sys\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.429420 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.429410    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-podres\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.429420 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.429419    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-proc\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.429684 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.429413    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-lib-modules\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.437490 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.437462    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnm77\" (UniqueName: \"kubernetes.io/projected/1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc-kube-api-access-lnm77\") pod \"perf-node-gather-daemonset-p7jcn\" (UID: \"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc\") " pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.597583 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.597488    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:21.724452 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:21.724427    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"]
Aug 01 00:50:21.727143 ip-10-0-131-136 kubenswrapper[2551]: W0801 00:50:21.727114    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1d2bc2d9_1bbf_46af_9ab6_4b226f4b6fcc.slice/crio-8115004cc570f6b7366f17dfc34b652e30aaf18dcc7efbd415023bdbc3007ca5 WatchSource:0}: Error finding container 8115004cc570f6b7366f17dfc34b652e30aaf18dcc7efbd415023bdbc3007ca5: Status 404 returned error can't find the container with id 8115004cc570f6b7366f17dfc34b652e30aaf18dcc7efbd415023bdbc3007ca5
Aug 01 00:50:22.096667 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:22.096627    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn" event={"ID":"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc","Type":"ContainerStarted","Data":"f1a4a92a7983feefb45282794ce3fbdb8129737cbdaca061834e89dfd5f81cf1"}
Aug 01 00:50:22.096882 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:22.096683    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn" event={"ID":"1d2bc2d9-1bbf-46af-9ab6-4b226f4b6fcc","Type":"ContainerStarted","Data":"8115004cc570f6b7366f17dfc34b652e30aaf18dcc7efbd415023bdbc3007ca5"}
Aug 01 00:50:22.096882 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:22.096766    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:22.111335 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:22.111274    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn" podStartSLOduration=1.1112572059999999 podStartE2EDuration="1.111257206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:50:22.111044988 +0000 UTC m=+3864.805092416" watchObservedRunningTime="2026-08-01 00:50:22.111257206 +0000 UTC m=+3864.805304625"
Aug 01 00:50:22.887255 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:22.887228    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bsrg5_97fb1913-5741-455f-8595-177262ce365a/dns/0.log"
Aug 01 00:50:22.906358 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:22.906339    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bsrg5_97fb1913-5741-455f-8595-177262ce365a/kube-rbac-proxy/0.log"
Aug 01 00:50:23.000848 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:23.000822    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k7f2q_818c8b69-f230-4bf7-b050-ae9e23c0281b/dns-node-resolver/0.log"
Aug 01 00:50:23.533886 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:23.533854    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-77b58c48d7-pdhnl_03207f34-03d6-4655-ba13-0de638a39a5d/registry/0.log"
Aug 01 00:50:23.600002 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:23.599970    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wkzlt_61b45fe3-b6e6-47f9-8974-11cb808bc9f0/node-ca/0.log"
Aug 01 00:50:24.696028 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:24.696001    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/0.log"
Aug 01 00:50:24.722935 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:24.722884    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/1.log"
Aug 01 00:50:25.307352 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:25.307325    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jxsn8_9ab1356e-7c27-49ae-a3ad-700b2c0ac399/serve-healthcheck-canary/0.log"
Aug 01 00:50:25.980906 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:25.980876    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h9q9h_f668c162-e18e-43ab-ad64-d2d290bf1301/kube-rbac-proxy/0.log"
Aug 01 00:50:26.032781 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:26.032752    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h9q9h_f668c162-e18e-43ab-ad64-d2d290bf1301/exporter/0.log"
Aug 01 00:50:26.089060 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:26.089018    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h9q9h_f668c162-e18e-43ab-ad64-d2d290bf1301/extractor/0.log"
Aug 01 00:50:28.110169 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:28.110136    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kbl7c/perf-node-gather-daemonset-p7jcn"
Aug 01 00:50:29.389166 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:29.389131    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cb8c69f9-4t7n9_cc5d85bc-955d-4cfa-88a4-7d036c8ffe20/manager/0.log"
Aug 01 00:50:29.415729 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:29.415696    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-nmzkm_737ff00b-2dbb-4d4d-b71a-43d49065c0f8/openshift-lws-operator/0.log"
Aug 01 00:50:30.479295 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:30.479257    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-786bd6bfd6-spzff_e0056bb9-f2c5-4115-abf3-6136f785c4a2/manager/0.log"
Aug 01 00:50:30.525909 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:30.525878    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/0.log"
Aug 01 00:50:30.552761 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:30.552734    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/1.log"
Aug 01 00:50:31.951324 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:31.951232    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/0.log"
Aug 01 00:50:31.980612 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:31.980573    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-6dzv7_cbeff71f-8a02-4ed2-94cd-aa1ced7671ac/istio-proxy/1.log"
Aug 01 00:50:37.007487 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:37.007454    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-86226_40e0e72a-edb8-4893-9b23-e12962884e30/kube-multus/0.log"
Aug 01 00:50:37.037486 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:37.037459    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-66n9v_502fbd41-3e97-4d6e-ac72-d6a05311d675/kube-multus-additional-cni-plugins/0.log"
Aug 01 00:50:37.058371 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:37.058345    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-66n9v_502fbd41-3e97-4d6e-ac72-d6a05311d675/egress-router-binary-copy/0.log"
Aug 01 00:50:37.078292 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:37.078267    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-66n9v_502fbd41-3e97-4d6e-ac72-d6a05311d675/cni-plugins/0.log"
Aug 01 00:50:37.099296 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:37.099276    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-66n9v_502fbd41-3e97-4d6e-ac72-d6a05311d675/bond-cni-plugin/0.log"
Aug 01 00:50:37.123347 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:37.123320    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-66n9v_502fbd41-3e97-4d6e-ac72-d6a05311d675/routeoverride-cni/0.log"
Aug 01 00:50:37.143560 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:37.143539    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-66n9v_502fbd41-3e97-4d6e-ac72-d6a05311d675/whereabouts-cni-bincopy/0.log"
Aug 01 00:50:37.163636 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:37.163612    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-66n9v_502fbd41-3e97-4d6e-ac72-d6a05311d675/whereabouts-cni/0.log"
Aug 01 00:50:37.662608 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:37.662579    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r7fbk_a0d57ecc-0561-4cce-855e-0281fb270d89/network-metrics-daemon/0.log"
Aug 01 00:50:37.679379 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:37.679339    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r7fbk_a0d57ecc-0561-4cce-855e-0281fb270d89/kube-rbac-proxy/0.log"
Aug 01 00:50:38.434164 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:38.434123    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b47km_84c96591-3b96-4722-a376-ea2b0728b358/ovn-controller/0.log"
Aug 01 00:50:38.448933 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:38.448866    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b47km_84c96591-3b96-4722-a376-ea2b0728b358/ovn-acl-logging/0.log"
Aug 01 00:50:38.468563 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:38.468540    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b47km_84c96591-3b96-4722-a376-ea2b0728b358/kube-rbac-proxy-node/0.log"
Aug 01 00:50:38.487243 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:38.487223    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b47km_84c96591-3b96-4722-a376-ea2b0728b358/kube-rbac-proxy-ovn-metrics/0.log"
Aug 01 00:50:38.504821 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:38.504799    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b47km_84c96591-3b96-4722-a376-ea2b0728b358/northd/0.log"
Aug 01 00:50:38.523593 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:38.523573    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b47km_84c96591-3b96-4722-a376-ea2b0728b358/nbdb/0.log"
Aug 01 00:50:38.543507 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:38.543488    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b47km_84c96591-3b96-4722-a376-ea2b0728b358/sbdb/0.log"
Aug 01 00:50:38.596883 ip-10-0-131-136 kubenswrapper[2551]: I0801 00:50:38.596861    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b47km_84c96591-3b96-4722-a376-ea2b0728b358/ovnkube-controller/0.log"